Nov 01 21:41:18.898986 ip-10-0-131-110 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 21:41:18.898997 ip-10-0-131-110 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 21:41:18.899004 ip-10-0-131-110 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 21:41:18.899259 ip-10-0-131-110 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 21:41:30.192011 ip-10-0-131-110 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 21:41:30.192028 ip-10-0-131-110 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f493d2a369774a8bab5b39484d6adca3 -- Nov 01 21:43:24.227709 ip-10-0-131-110 systemd[1]: Starting Kubernetes Kubelet... Nov 01 21:43:24.652808 ip-10-0-131-110 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:24.652808 ip-10-0-131-110 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 21:43:24.652808 ip-10-0-131-110 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:24.652808 ip-10-0-131-110 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:24.656258 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656178 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.656258 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656258 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656262 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656265 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656268 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656271 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656274 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656277 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656279 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656282 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656284 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656287 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656291 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656294 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656298 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656302 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656306 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656310 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656314 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656316 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.656327 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656319 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656327 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656330 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656336 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656339 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656342 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656345 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656348 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656350 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656353 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656355 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656358 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656361 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656364 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656367 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656369 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656372 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.656872 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656377 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656380 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656382 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656387 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656390 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656392 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656394 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656397 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656400 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656403 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656405 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656408 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656411 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656414 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656417 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656419 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656422 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656424 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.657320 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656427 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656429 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656434 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656437 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656439 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656455 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656458 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656461 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656463 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656466 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656468 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656471 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656474 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656477 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656480 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656482 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656485 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656489 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656492 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.657789 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656494 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656497 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656500 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656503 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656506 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656509 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656511 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656515 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656520 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656523 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656526 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656528 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656531 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656534 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656537 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656539 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656542 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656544 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656547 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.658249 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656549 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656553 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656555 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656558 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656561 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656563 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656566 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.656569 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657411 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657419 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657422 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657425 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657428 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657432 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657437 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657456 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657460 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657463 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657466 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.658746 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657469 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657472 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657475 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657478 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657481 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657483 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657486 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657489 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657491 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657494 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657497 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657499 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657502 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657504 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657511 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657515 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657518 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657521 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657523 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.659215 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657527 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657537 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657540 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657543 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657546 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657549 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657552 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657555 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657557 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657560 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657562 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657565 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657568 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657570 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657573 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657575 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657578 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657580 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657583 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.659701 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657585 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657588 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657590 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657593 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657595 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657598 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657600 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657603 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657605 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657608 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657610 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657613 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657615 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657618 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657620 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657623 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657626 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657629 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657631 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657636 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.660178 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657638 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657641 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657643 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657646 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657649 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657651 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657654 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657656 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657659 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657661 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657664 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657666 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657669 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657671 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657674 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657676 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657679 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657681 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657684 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.660662 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657686 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657689 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657692 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657694 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657697 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657699 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657701 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657704 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657707 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657709 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657713 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657715 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657718 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657721 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657725 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657727 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.657730 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658270 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658276 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658280 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.661109 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658282 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658285 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658288 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658291 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658293 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658296 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658304 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658309 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658313 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658324 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658327 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658330 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658333 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.661612 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658335 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658338 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658341 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658349 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658353 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658356 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658359 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658361 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658364 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658367 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658370 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658372 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658375 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658378 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658380 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658383 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658385 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658388 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658390 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658393 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.662042 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658396 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658398 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658401 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658404 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658406 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658409 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658411 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658414 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658417 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658419 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658421 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658424 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658426 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658429 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658431 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658434 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658438 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658455 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658458 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.662571 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658461 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658463 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658468 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658471 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658474 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658477 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658480 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658482 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658485 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658488 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658490 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658492 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658495 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658498 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658501 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658504 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658507 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658509 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658512 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658515 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.663014 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658517 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658520 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658522 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658525 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658528 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658533 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658536 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658538 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658541 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658544 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658546 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658549 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658552 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658563 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658566 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658571 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658573 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.663493 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658576 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658578 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658581 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658584 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658586 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658589 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.658593 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659592 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659601 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659608 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659613 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659617 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659620 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659624 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659628 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659632 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659635 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659638 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659642 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659645 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659648 2578 flags.go:64] FLAG: --cgroup-root="" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659651 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659654 2578 flags.go:64] FLAG: --client-ca-file="" Nov 01 21:43:24.663978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659657 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659660 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659665 2578 flags.go:64] FLAG: --cluster-domain="" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659668 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659671 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659674 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659677 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659682 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659684 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659687 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659691 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659694 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659697 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659700 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659703 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659707 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659711 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659715 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659718 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659721 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659724 2578 flags.go:64] FLAG: --enable-server="true" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659727 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659731 2578 flags.go:64] FLAG: --event-burst="100" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659734 2578 flags.go:64] FLAG: --event-qps="50" Nov 01 21:43:24.664567 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659737 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659740 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659743 2578 flags.go:64] FLAG: --eviction-hard="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659747 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659750 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659753 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659757 2578 flags.go:64] FLAG: --eviction-soft="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659760 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659763 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659766 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659775 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659778 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659781 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659784 2578 flags.go:64] FLAG: --feature-gates="" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659788 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659792 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659795 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659798 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659801 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659804 2578 flags.go:64] FLAG: --help="false" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659807 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659810 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659813 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 21:43:24.665163 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659816 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659820 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659823 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659826 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659829 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659832 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659835 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659838 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659841 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659844 2578 flags.go:64] FLAG: --kube-reserved="" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659847 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659850 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659853 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659856 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659859 2578 flags.go:64] FLAG: --lock-file="" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659862 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659865 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659868 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659874 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659877 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659886 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659889 2578 flags.go:64] FLAG: --logging-format="text" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659892 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659895 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 21:43:24.665747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659898 2578 flags.go:64] FLAG: --manifest-url="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659901 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659905 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659908 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659912 2578 flags.go:64] FLAG: --max-pods="110" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659915 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659918 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659921 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659924 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659927 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659930 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659933 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659939 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659942 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659945 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659948 2578 flags.go:64] FLAG: --pod-cidr="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659951 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659956 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659968 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659971 2578 flags.go:64] FLAG: --port="10250" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659974 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659977 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09bd62f6425b1bfb0" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659980 2578 flags.go:64] FLAG: --qos-reserved="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659983 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659986 2578 flags.go:64] FLAG: --register-node="true" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659989 2578 flags.go:64] FLAG: --register-with-taints="" Nov 01 21:43:24.666317 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659995 2578 flags.go:64] FLAG: --registry-burst="10" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.659998 2578 flags.go:64] FLAG: --registry-qps="5" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660001 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660009 2578 flags.go:64] FLAG: --reserved-memory="" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660013 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660016 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660019 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660022 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660025 2578 flags.go:64] FLAG: --runonce="false" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660028 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660031 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660034 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660037 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660040 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660043 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660046 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660049 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660052 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660055 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660058 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660060 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660063 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660066 2578 flags.go:64] FLAG: --system-cgroups="" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660071 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660076 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 21:43:24.666970 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660079 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660082 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660086 2578 flags.go:64] FLAG: --tls-min-version="" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660088 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660091 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660094 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660097 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660100 2578 flags.go:64] FLAG: --v="2" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660104 2578 flags.go:64] FLAG: --version="false" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660108 2578 flags.go:64] FLAG: --vmodule="" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660112 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660121 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660308 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660312 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660315 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660317 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660322 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660325 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660327 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660330 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660332 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660335 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.667602 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660338 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660340 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660343 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660345 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660348 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660351 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660353 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660356 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660366 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660368 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660371 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660374 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660377 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660379 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660382 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660387 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.668126 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660391 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660395 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660397 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660407 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660410 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660412 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660415 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660419 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660421 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660424 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660426 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660429 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660434 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660437 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660439 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660453 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660456 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660458 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.668590 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660461 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660464 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660468 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660470 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660475 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660478 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660482 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660485 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660488 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660490 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660493 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660499 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660502 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660505 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660507 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660510 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660521 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660524 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660526 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.669056 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660530 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660533 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660536 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660538 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660541 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660543 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660546 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660549 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660551 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660554 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660556 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660559 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660561 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660564 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660571 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660573 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660576 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660578 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660581 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660583 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.669554 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660586 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660589 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660591 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660594 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660596 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660599 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660601 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660604 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660606 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660609 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660617 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660620 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660624 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660626 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660629 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660631 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660634 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.670033 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660636 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.660663 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660722 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660727 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660731 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660735 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660739 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660743 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660746 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660751 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660754 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660756 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660759 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660762 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660764 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660767 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.670433 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660769 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660772 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660775 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660777 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660780 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660782 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660785 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660788 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660790 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660793 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660801 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660804 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660808 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660810 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660813 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660816 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660818 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660821 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660823 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.670939 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660826 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660829 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660831 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660834 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660836 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660839 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660843 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660846 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660848 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660851 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660853 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660856 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660858 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660861 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660863 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660866 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660869 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660871 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660874 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660876 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.671393 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660879 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660881 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660884 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660897 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660905 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660909 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660912 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660914 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660917 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660919 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660922 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660925 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660927 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660930 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660932 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660935 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660938 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660940 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660944 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.671882 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660949 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660952 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660957 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660960 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660962 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660965 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660967 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660970 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660973 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660975 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660978 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660981 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660985 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660989 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660992 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.660995 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661004 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661007 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.672337 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661010 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661012 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661015 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661017 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661020 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661022 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661025 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661033 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661036 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661039 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661041 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661045 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661048 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.661050 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.669237 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.669252 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.670075 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.670082 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670140 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670145 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.672796 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670148 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670151 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670154 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670157 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670160 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670163 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670165 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670169 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670171 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670174 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670177 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670180 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670183 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670185 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670188 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670190 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670193 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670197 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670202 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670206 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.673322 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670210 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670215 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670219 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670222 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670225 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670228 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670231 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670235 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670238 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670240 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670243 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670246 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670249 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670252 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670254 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670257 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670259 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670262 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670264 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670267 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.673819 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670269 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670272 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670274 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670278 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670280 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670283 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670285 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670288 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670290 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670293 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670295 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670300 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670303 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670305 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670308 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670310 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670313 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670316 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.674291 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670318 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670324 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670326 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670329 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670332 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670336 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670339 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670341 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670346 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670349 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670351 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670354 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670356 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670359 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670361 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670364 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670367 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670369 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.674749 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670372 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670377 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670379 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670382 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670384 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670387 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670391 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670395 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670399 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670401 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670404 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670407 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670410 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670413 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670416 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670418 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670421 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670423 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.675220 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670426 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670428 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670430 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670433 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670436 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670438 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.670477 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670616 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670621 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670624 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670627 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670630 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670633 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670636 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670638 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670641 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.675674 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670644 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670646 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670649 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670652 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670654 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670657 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670662 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670665 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670667 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670670 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670673 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670675 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670678 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670681 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670683 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670686 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670688 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670691 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.676063 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670693 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670695 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670698 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670700 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670703 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670705 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670708 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670710 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670713 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670716 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670719 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670721 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670723 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670726 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670729 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670732 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670734 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670737 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670739 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670742 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.676613 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670745 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670747 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670750 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670754 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670756 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670759 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670761 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670764 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670766 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670768 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670771 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670773 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670776 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670781 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670783 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670786 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670788 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670791 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670793 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.677081 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670795 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670798 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670801 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670805 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670808 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670811 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670814 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670816 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670820 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670824 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670827 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670830 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670833 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670835 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670838 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670840 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670843 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670846 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670848 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.677557 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670851 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670853 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670856 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670858 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670861 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670863 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670866 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670868 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670871 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670873 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670875 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670878 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670880 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670883 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670885 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670888 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670891 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670893 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.677998 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670896 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.670919 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670953 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670958 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670961 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670964 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670967 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670970 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670972 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670975 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670978 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670980 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670983 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670985 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670988 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670990 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:24.678416 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670993 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.670998 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671001 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671003 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671006 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671008 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671011 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671013 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671016 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671018 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671021 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671023 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671026 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671028 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671031 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671034 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671037 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671039 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671042 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:24.678800 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671044 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671047 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671049 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671052 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671054 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671057 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671059 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671062 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671065 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671067 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671070 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671072 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671075 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671078 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671080 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671083 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671086 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671088 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671090 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:24.679259 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671093 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671095 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671098 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671100 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671103 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671105 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671108 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671111 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671113 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671116 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671118 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671121 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671123 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671126 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671128 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671131 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671136 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671138 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671141 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:24.679886 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671146 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671149 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671151 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671154 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671157 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671160 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671162 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671165 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671168 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671170 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671173 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671175 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671177 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671180 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671182 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671187 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671190 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671192 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671197 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:24.680712 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671200 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671203 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671206 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671208 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671211 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671213 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671215 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671219 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671226 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671228 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671231 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:24.671236 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.671283 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.676235 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.677317 2578 server.go:1014] "Starting client certificate rotation" Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.677422 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:43:24.681518 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.677466 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:43:24.682085 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.681169 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:43:24.682648 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.682632 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:43:24.695941 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.695921 2578 log.go:25] "Validated CRI v1 runtime API" Nov 01 21:43:24.700906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.700892 2578 log.go:25] "Validated CRI v1 image API" Nov 01 21:43:24.703137 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.703110 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 21:43:24.707630 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.707610 2578 fs.go:135] Filesystem UUIDs: map[116cfc5e-23a0-4d66-8912-851450b0b085:/dev/nvme0n1p4 79bda8b1-d733-4548-9b93-0579954c339e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 21:43:24.707688 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.707630 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 21:43:24.713598 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.713483 2578 manager.go:217] Machine: {Timestamp:2026-11-01 21:43:24.712132882 +0000 UTC m=+0.376415136 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100631 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec257749b6a70b71afd90a35ad161251 SystemUUID:ec257749-b6a7-0b71-afd9-0a35ad161251 BootID:f493d2a3-6977-4a8b-ab5b-39484d6adca3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:f6:24:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:f6:24:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:01:b7:cf:99:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 21:43:24.714259 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.714249 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 21:43:24.714340 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.714329 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 21:43:24.715568 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.715552 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 21:43:24.718046 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.718025 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 21:43:24.718190 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.718050 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-110.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 21:43:24.718236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.718192 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 21:43:24.718236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.718201 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 21:43:24.718236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.718211 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:43:24.719129 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.719119 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:43:24.720327 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.720317 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 21:43:24.725008 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.724996 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 21:43:24.725134 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725124 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 21:43:24.725162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725157 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 01 21:43:24.725193 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725172 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 21:43:24.725193 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725181 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 21:43:24.725193 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725190 2578 kubelet.go:405] "Adding apiserver pod source" Nov 01 21:43:24.725284 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725198 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 21:43:24.725480 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.725469 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:43:24.729292 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.729272 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 21:43:24.734829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.734817 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:43:24.735784 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.735771 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 21:43:24.735823 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.735804 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 21:43:24.736552 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736539 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 21:43:24.736600 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736555 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 21:43:24.736600 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736561 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 21:43:24.736600 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736567 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 21:43:24.736600 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736573 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 21:43:24.736600 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.736578 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 21:43:24.737250 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737241 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 21:43:24.737277 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737251 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 21:43:24.737277 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737258 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 21:43:24.737277 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737263 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 21:43:24.737277 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737272 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 21:43:24.737397 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.737280 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 21:43:24.738206 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.738197 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 21:43:24.738206 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.738207 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 21:43:24.742084 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.742070 2578 server.go:1263] "Started kubelet" Nov 01 21:43:24.742207 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.742151 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 21:43:24.742252 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.742196 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 21:43:24.742288 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.742271 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 21:43:24.742861 ip-10-0-131-110 systemd[1]: Started Kubernetes Kubelet. Nov 01 21:43:24.744033 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.744015 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 21:43:24.744945 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.744884 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 01 21:43:24.745617 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.745584 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 21:43:24.745617 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.745591 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-110.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 21:43:24.745767 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.745678 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-110.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 21:43:24.748648 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.748615 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 21:43:24.752948 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.752934 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 21:43:24.753013 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.752955 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 21:43:24.753072 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.753010 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:24.753656 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753641 2578 factory.go:55] Registering systemd factory Nov 01 21:43:24.753656 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753656 2578 factory.go:223] Registration of the systemd container factory successfully Nov 01 21:43:24.753772 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753681 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 21:43:24.753849 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753826 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 21:43:24.753932 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753914 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 21:43:24.753932 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753929 2578 factory.go:153] Registering CRI-O factory Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753940 2578 factory.go:223] Registration of the crio container factory successfully Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.753983 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.754000 2578 factory.go:103] Registering Raw factory Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.754017 2578 manager.go:1199] Started watching for new ooms in manager Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.754023 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 01 21:43:24.754099 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.754042 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 01 21:43:24.754781 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.753577 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.us-east-2.compute.internal.18e409def1a856bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 21:43:24.742031037 +0000 UTC m=+0.406313272,LastTimestamp:2026-11-01 21:43:24.742031037 +0000 UTC m=+0.406313272,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.us-east-2.compute.internal,}" Nov 01 21:43:24.754994 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.754981 2578 manager.go:319] Starting recovery of all containers Nov 01 21:43:24.759697 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.759668 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-110.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 21:43:24.765641 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.765625 2578 manager.go:324] Recovery completed Nov 01 21:43:24.769135 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.769111 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 21:43:24.769797 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.769785 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:24.772018 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.772000 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fsp7m" Nov 01 21:43:24.773002 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.772981 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:24.773068 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773021 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:24.773068 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773039 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:24.773598 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773583 2578 cpu_manager.go:226] "Starting" policy="none" Nov 01 21:43:24.773598 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773593 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 21:43:24.773738 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773603 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:24.773738 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.773611 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 21:43:24.774401 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.774385 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:24.774438 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.774420 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:24.774483 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.774461 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:24.775130 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.775065 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.us-east-2.compute.internal.18e409def380eaf1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-110.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 21:43:24.773001969 +0000 UTC m=+0.437284207,LastTimestamp:2026-11-01 21:43:24.773001969 +0000 UTC m=+0.437284207,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.us-east-2.compute.internal,}" Nov 01 21:43:24.775781 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.775770 2578 policy_none.go:50] "Start" Nov 01 21:43:24.775829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.775784 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 21:43:24.775829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.775796 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 21:43:24.777756 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.777745 2578 policy_none.go:44] "Start" Nov 01 21:43:24.785003 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.784985 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fsp7m" Nov 01 21:43:24.787144 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.787057 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.us-east-2.compute.internal.18e409def3815a4d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-110.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 21:43:24.773030477 +0000 UTC m=+0.437312717,LastTimestamp:2026-11-01 21:43:24.773030477 +0000 UTC m=+0.437312717,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.us-east-2.compute.internal,}" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.805991 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809271 2578 manager.go:341] "Starting Device Plugin manager" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.809297 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809305 2578 server.go:87] "Starting device plugin registration server" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809504 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809513 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809625 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809746 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.809756 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.810158 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.810205 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.826540 2578 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2616 into the system container \"/system.slice\": " Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.910474 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.913124 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.913155 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.913168 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:24.917599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.913199 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:24.922978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.922959 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:24.923082 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.922986 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-110.us-east-2.compute.internal\": node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:24.932482 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.932338 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 21:43:24.933568 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.933548 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 21:43:24.933646 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.933570 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 21:43:24.933646 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.933591 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 21:43:24.933646 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.933625 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 21:43:24.938164 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:24.938143 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:24.943762 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:24.943746 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.034676 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.034652 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal"] Nov 01 21:43:25.034744 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.034727 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:25.035488 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.035467 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:25.035569 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.035499 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:25.035569 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.035510 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:25.037750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.037738 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:25.037889 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.037875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.037934 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.037905 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038656 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038531 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038711 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038723 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038690 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:25.038911 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.038892 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:25.041152 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.041132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.041239 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.041158 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:25.041859 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.041838 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:25.041935 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.041874 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:25.041935 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.041889 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:25.043833 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.043813 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.055945 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.055930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.056027 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.055952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.056027 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.055971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.074752 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.074732 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.079213 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.079197 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.144005 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.143988 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.156316 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.156366 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.156366 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.156428 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.156428 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.156507 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.156423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.244759 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.244704 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.345200 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.345183 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.378745 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.378726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.382973 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.382959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:25.445904 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.445879 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.546412 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.546353 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.646915 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.646894 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.676839 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.676818 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 21:43:25.677185 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.676939 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 21:43:25.747012 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.746985 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.792802 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.792754 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:25 +0000 UTC" deadline="2028-04-03 21:36:58.638305132 +0000 UTC" Nov 01 21:43:25.792892 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.792799 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12455h53m32.845512568s" Nov 01 21:43:25.825333 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.825280 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:25.847566 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.847544 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:25.925426 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.925397 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:25.934485 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.934463 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:25.935728 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.935707 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:25.935822 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.935755 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:25.935822 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:25.935772 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:25.947677 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:25.947654 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:26.043240 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:26.043208 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571993d965c8e1c01034d44492f7be15.slice/crio-f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e WatchSource:0}: Error finding container f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e: Status 404 returned error can't find the container with id f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e Nov 01 21:43:26.043636 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:26.043606 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8a4eb4513497bb3852f6aab7c20b22.slice/crio-53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2 WatchSource:0}: Error finding container 53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2: Status 404 returned error can't find the container with id 53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2 Nov 01 21:43:26.048783 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.048762 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:43:26.051486 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.051472 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:43:26.060571 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.060556 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:26.154008 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.153961 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:26.166073 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.166052 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:43:26.167249 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.167233 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:43:26.180889 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.180870 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:43:26.728852 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.728825 2578 apiserver.go:52] "Watching apiserver" Nov 01 21:43:26.737706 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.737684 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 21:43:26.738104 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.738077 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ckgds","kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-8n2mc","openshift-image-registry/node-ca-q2cpx","openshift-network-diagnostics/network-check-target-t7lmc","openshift-network-operator/iptables-alerter-z9t47","openshift-ovn-kubernetes/ovnkube-node-lcz57","kube-system/global-pull-secret-syncer-wr8bc","openshift-dns/node-resolver-jxl58","openshift-multus/multus-additional-cni-plugins-4tw88","openshift-multus/network-metrics-daemon-vlnwn","kube-system/konnectivity-agent-4tvq4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm"] Nov 01 21:43:26.740817 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.740788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.743153 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.743133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.743256 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.743209 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:26.745187 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.745157 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 21:43:26.745358 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.745257 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\"" Nov 01 21:43:26.745834 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.745816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.746719 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.746696 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 21:43:26.746824 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.746778 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.746887 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.746842 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 21:43:26.746953 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.746942 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.747212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.747189 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 21:43:26.751603 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.749908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.753246 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.752943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:26.753246 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.753034 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:26.755188 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.754357 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 21:43:26.756487 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.756465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.761228 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.761207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.765056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.764313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nbl\" (UniqueName: \"kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skmp\" (UniqueName: \"kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxr9h\" (UniqueName: \"kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.766857 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.766985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.767665 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxjq\" (UniqueName: \"kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdbf\" (UniqueName: \"kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.767609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.768396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.768585 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.768456 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:26.771123 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.771098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.772281 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.772052 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 21:43:26.778138 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.778001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.780723 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.780668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.787057 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.787037 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fqr7" Nov 01 21:43:26.799935 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.799916 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fqr7" Nov 01 21:43:26.854840 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.854821 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 21:43:26.855232 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855209 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.855368 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855350 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\"" Nov 01 21:43:26.855446 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855390 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 21:43:26.855624 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855500 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.855718 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855683 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\"" Nov 01 21:43:26.855791 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855710 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\"" Nov 01 21:43:26.855791 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855779 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 21:43:26.855909 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855814 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 21:43:26.855909 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 21:43:26.855909 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855888 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.856050 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855994 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\"" Nov 01 21:43:26.856050 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856006 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.856050 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856023 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 21:43:26.856165 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856065 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 21:43:26.856165 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856084 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.856165 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856112 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.856257 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856173 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\"" Nov 01 21:43:26.856338 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856315 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.856430 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856359 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.856496 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856448 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.856496 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856488 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.856594 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.856579 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 21:43:26.857628 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.857593 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 21:43:26.857721 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.857655 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 21:43:26.857721 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.857714 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 21:43:26.857871 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.857744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\"" Nov 01 21:43:26.857871 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.855702 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 21:43:26.858380 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.858364 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\"" Nov 01 21:43:26.858495 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.858475 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\"" Nov 01 21:43:26.867843 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867811 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.867955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.867955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxjq\" (UniqueName: \"kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.867955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.867955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rjb\" (UniqueName: \"kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.867993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7v9\" (UniqueName: \"kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfpw\" (UniqueName: \"kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868172 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nbl\" (UniqueName: \"kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skmp\" (UniqueName: \"kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr9h\" (UniqueName: \"kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.868671 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.868978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869039 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 21:43:26.869291 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdbf\" (UniqueName: \"kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4c9c\" (UniqueName: \"kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.870045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.870894 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.869961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870458 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.870654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871426 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.871705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872452 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.871786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.872452 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872432 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.872592 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872592 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.872716 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.872776 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872936 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.872936 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.873040 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.872962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.873294 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.873271 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.873544 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.873503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.873770 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.873747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.873840 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.873754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.873901 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.873836 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:26.873901 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.873896 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:27.373877599 +0000 UTC m=+3.038159842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874131 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874179 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.874364 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.874228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.884142 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.884098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.889987 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.889043 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:26.889987 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.889069 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:26.889987 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.889084 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:26.889987 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.889148 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:27.389131253 +0000 UTC m=+3.053413494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:26.892747 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.892722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skmp\" (UniqueName: \"kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:26.895883 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.895840 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nbl\" (UniqueName: \"kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:26.896344 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.896312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxjq\" (UniqueName: \"kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:26.899520 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.899474 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdbf\" (UniqueName: \"kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:26.900045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.899997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr9h\" (UniqueName: \"kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:26.940385 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.940326 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerStarted","Data":"f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e"} Nov 01 21:43:26.942371 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.942344 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"fc8a4eb4513497bb3852f6aab7c20b22","Type":"ContainerStarted","Data":"53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2"} Nov 01 21:43:26.970207 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfpw\" (UniqueName: \"kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.970326 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970391 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970391 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970517 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970517 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.970517 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.970517 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970648 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.970729 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4c9c\" (UniqueName: \"kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.970974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rjb\" (UniqueName: \"kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7v9\" (UniqueName: \"kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.971750 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.971947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972146 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.972280 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:26.972444 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:26.972357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:27.472339964 +0000 UTC m=+3.136622188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:26.972887 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.972887 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972887 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.972887 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.973105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.973105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.972988 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.973105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.973105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.973105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.973360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.973360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:26.973360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:26.973360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.973674 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.973827 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.973809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:26.983756 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.983404 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfpw\" (UniqueName: \"kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:26.983756 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.983692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4c9c\" (UniqueName: \"kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:26.985943 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.985920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rjb\" (UniqueName: \"kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:26.987749 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:26.987725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7v9\" (UniqueName: \"kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:43:27.076734 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.076708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:43:27.079398 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.079378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:43:27.087663 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.087642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:27.088400 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.088378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:43:27.089533 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.089498 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode632f276_6eef_4478_8c2c_1f7e4b98a6d7.slice/crio-6064c4aa0ebcc9e8c3a442a46824ed13768d645c7e7225afe1410a9746779be7 WatchSource:0}: Error finding container 6064c4aa0ebcc9e8c3a442a46824ed13768d645c7e7225afe1410a9746779be7: Status 404 returned error can't find the container with id 6064c4aa0ebcc9e8c3a442a46824ed13768d645c7e7225afe1410a9746779be7 Nov 01 21:43:27.097511 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.097490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jxl58" Nov 01 21:43:27.109823 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.109801 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d6f94c_7d2b_420b_bea6_0bb929dd6194.slice/crio-f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d WatchSource:0}: Error finding container f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d: Status 404 returned error can't find the container with id f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d Nov 01 21:43:27.115236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.115162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:43:27.125922 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.125879 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5cff7d_1650_41de_9984_fd5cbd4b3fea.slice/crio-3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806 WatchSource:0}: Error finding container 3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806: Status 404 returned error can't find the container with id 3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806 Nov 01 21:43:27.131059 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.131032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckgds" Nov 01 21:43:27.133214 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.133191 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:43:27.136078 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.136056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:27.143037 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.143002 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403cfda3_7bce_4dfa_bb72_b7f9f26568d0.slice/crio-4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905 WatchSource:0}: Error finding container 4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905: Status 404 returned error can't find the container with id 4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905 Nov 01 21:43:27.150046 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.150022 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb0e1e5_b593_4c2b_9950_ae9c967710e1.slice/crio-c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a WatchSource:0}: Error finding container c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a: Status 404 returned error can't find the container with id c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a Nov 01 21:43:27.150924 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:27.150902 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb4e811_66fe_409f_8a75_de7e9e9f936f.slice/crio-329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6 WatchSource:0}: Error finding container 329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6: Status 404 returned error can't find the container with id 329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6 Nov 01 21:43:27.269932 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.269857 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:27.374136 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.374050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:27.374318 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.374238 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:27.374391 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.374320 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:28.374298606 +0000 UTC m=+4.038580844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:27.476046 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.476011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:27.476223 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.476059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:27.476223 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476204 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:27.476420 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476222 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:27.476420 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476246 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:27.476420 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476260 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:27.476420 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476281 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:28.476260962 +0000 UTC m=+4.140543198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:27.476420 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.476313 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:28.476295853 +0000 UTC m=+4.140578089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:27.804253 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.804125 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:26 +0000 UTC" deadline="2028-08-06 21:43:54.51650396 +0000 UTC" Nov 01 21:43:27.804253 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.804186 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15456h0m26.712322109s" Nov 01 21:43:27.934745 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.934716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:27.934905 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:27.934816 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:27.952662 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.952623 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9t47" event={"ID":"36c671f8-9973-4f01-a790-c286bb2ee732","Type":"ContainerStarted","Data":"fb8da50ec4f893be99d3fb74d9614ca4d46282180539786f497bbc54eb64ae85"} Nov 01 21:43:27.961090 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.961056 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" event={"ID":"e632f276-6eef-4478-8c2c-1f7e4b98a6d7","Type":"ContainerStarted","Data":"6064c4aa0ebcc9e8c3a442a46824ed13768d645c7e7225afe1410a9746779be7"} Nov 01 21:43:27.964753 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.964725 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckgds" event={"ID":"403cfda3-7bce-4dfa-bb72-b7f9f26568d0","Type":"ContainerStarted","Data":"4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905"} Nov 01 21:43:27.966590 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.966530 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" event={"ID":"0b5cff7d-1650-41de-9984-fd5cbd4b3fea","Type":"ContainerStarted","Data":"3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806"} Nov 01 21:43:27.973597 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.973533 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jxl58" event={"ID":"89d6f94c-7d2b-420b-bea6-0bb929dd6194","Type":"ContainerStarted","Data":"f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d"} Nov 01 21:43:27.979012 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.978983 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4tvq4" event={"ID":"3cb4e811-66fe-409f-8a75-de7e9e9f936f","Type":"ContainerStarted","Data":"329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6"} Nov 01 21:43:27.986347 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.986320 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerStarted","Data":"c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a"} Nov 01 21:43:27.989124 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.989099 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2cpx" event={"ID":"8e6c0a65-773b-45c9-b18b-bd582d7a6307","Type":"ContainerStarted","Data":"9fab3e0e022816a4226308666f7df3d80173a21c9b52599044c53c0d70be9044"} Nov 01 21:43:27.995209 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:27.995182 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"7f2c9646582cdb9fe72f01372df1dc23c46b5fd1e78c99ce758ae622ad299510"} Nov 01 21:43:28.383868 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.383634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:28.383868 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.383761 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:28.383868 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.383810 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:30.383797114 +0000 UTC m=+6.048079351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.484954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.485016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485196 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485214 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485229 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:30.485274747 +0000 UTC m=+6.149556988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485744 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:28.485844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.485801 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:30.485784472 +0000 UTC m=+6.150066710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:28.804771 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.804673 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:26 +0000 UTC" deadline="2028-04-02 22:12:20.266595015 +0000 UTC" Nov 01 21:43:28.804771 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.804710 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12432h28m51.461889731s" Nov 01 21:43:28.936278 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.936248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:28.936428 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.936376 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:28.936807 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:28.936786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:28.936923 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:28.936902 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:29.933989 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:29.933743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:29.934506 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:29.934099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:30.015424 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.015386 2578 generic.go:356] "Generic (PLEG): container finished" podID="571993d965c8e1c01034d44492f7be15" containerID="f1e58b37b7c135d9cdce6b409e21deabb962d93d346b54e14ed36c7fa228cc71" exitCode=0 Nov 01 21:43:30.016373 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.016323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerDied","Data":"f1e58b37b7c135d9cdce6b409e21deabb962d93d346b54e14ed36c7fa228cc71"} Nov 01 21:43:30.028138 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.028113 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"fc8a4eb4513497bb3852f6aab7c20b22","Type":"ContainerStarted","Data":"b717c1abf4bdbecf6d589c21838d2dd3df18db069e53dbe85af787e6a880a435"} Nov 01 21:43:30.404946 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.404910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:30.405128 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.405057 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:30.405128 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.405121 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:34.405102463 +0000 UTC m=+10.069384699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:30.505779 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.505752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:30.505901 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.505800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:30.505962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.505908 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:30.505962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.505944 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:30.506049 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.505962 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:30.506049 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.505972 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:34.505954173 +0000 UTC m=+10.170236399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:30.506049 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.505974 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:30.506049 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.506027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:34.506011321 +0000 UTC m=+10.170293561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:30.936532 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.936460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:30.936979 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.936606 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:30.937036 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:30.936999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:30.937131 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:30.937092 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:31.041094 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:31.041056 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="246a9a09b8ad9def32e3c45e57ac4df01027d357893d3d4cd7d1d99b23a40566" exitCode=0 Nov 01 21:43:31.041269 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:31.041151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"246a9a09b8ad9def32e3c45e57ac4df01027d357893d3d4cd7d1d99b23a40566"} Nov 01 21:43:31.048492 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:31.048467 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerStarted","Data":"35b6d9fe4f4ddee886fd33e809fb87886ffeb7ce6537dd07c9ec155d7ec84eff"} Nov 01 21:43:31.068398 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:31.067091 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" podStartSLOduration=5.067074467 podStartE2EDuration="5.067074467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:30.057317938 +0000 UTC m=+5.721600183" watchObservedRunningTime="2026-11-01 21:43:31.067074467 +0000 UTC m=+6.731356712" Nov 01 21:43:31.934314 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:31.934280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:31.934570 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:31.934417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:32.055931 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.055876 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4tvq4" event={"ID":"3cb4e811-66fe-409f-8a75-de7e9e9f936f","Type":"ContainerStarted","Data":"73f474ab00f8c7fb2112f585d22f1b9c50a2b8963680c0b2dd83cb584b10fce3"} Nov 01 21:43:32.072437 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.072384 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" podStartSLOduration=6.072367231 podStartE2EDuration="6.072367231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:31.086548797 +0000 UTC m=+6.750831042" watchObservedRunningTime="2026-11-01 21:43:32.072367231 +0000 UTC m=+7.736649475" Nov 01 21:43:32.688568 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.688317 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:32.689009 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.688982 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:32.704895 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.704843 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4tvq4" podStartSLOduration=3.446724058 podStartE2EDuration="7.70482605s" totalImagesPullingTime="4.258101992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:32.073014219 +0000 UTC m=+7.737296462" watchObservedRunningTime="2026-11-01 21:43:32.70482605 +0000 UTC m=+8.369108295" Nov 01 21:43:32.936761 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.936734 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:32.936924 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:32.936838 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:32.937173 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:32.937155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:32.937272 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:32.937221 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:33.064564 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:33.064530 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2cpx" event={"ID":"8e6c0a65-773b-45c9-b18b-bd582d7a6307","Type":"ContainerStarted","Data":"9759e0e4e32291c2904fe51b29a66b2ed7f0519d77addd4a5e5c9002bd27bb07"} Nov 01 21:43:33.081576 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:33.081532 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2cpx" podStartSLOduration=2.967274673 podStartE2EDuration="8.08151325s" totalImagesPullingTime="5.114238577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:33.081103283 +0000 UTC m=+8.745385526" watchObservedRunningTime="2026-11-01 21:43:33.08151325 +0000 UTC m=+8.745795493" Nov 01 21:43:33.934054 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:33.934019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:33.934231 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:33.934152 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:34.068252 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.068221 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" event={"ID":"0b5cff7d-1650-41de-9984-fd5cbd4b3fea","Type":"ContainerStarted","Data":"d65c09bad59b726291a98d5f64476188f97b834117bbcfe45f3570bbdf806c10"} Nov 01 21:43:34.068641 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.068323 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:34.438769 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.438196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:34.438769 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.438350 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:34.438769 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.438416 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.438397132 +0000 UTC m=+18.102679359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.539203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.539258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539392 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539409 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539422 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539495 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.539477312 +0000 UTC m=+18.203759550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539883 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:34.539962 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.539929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.539914564 +0000 UTC m=+18.204196802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:34.935027 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.934971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:34.935167 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.935076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:34.935344 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.935328 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:34.935488 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:34.935433 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:34.971397 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:34.971365 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 21:43:35.071921 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.071837 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckgds" event={"ID":"403cfda3-7bce-4dfa-bb72-b7f9f26568d0","Type":"ContainerStarted","Data":"e759ddf206c4acab248089f8896e6889c6939a5bc6c6785ba00b8d4aaa520531"} Nov 01 21:43:35.082914 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.081569 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" event={"ID":"0b5cff7d-1650-41de-9984-fd5cbd4b3fea","Type":"ContainerStarted","Data":"26baf993bb86fa35ad14e327f4800fdabb55c279af8298879d12009f485bd0bd"} Nov 01 21:43:35.815586 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.815255 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:43:34.971383572Z","UUID":"15a675c1-afc6-42ba-b439-2762135da484","Handler":null,"Name":"","Endpoint":""} Nov 01 21:43:35.816955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.816930 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 21:43:35.816955 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.816960 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 21:43:35.934547 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:35.934045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:35.934547 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:35.934164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:36.086783 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.086695 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jxl58" event={"ID":"89d6f94c-7d2b-420b-bea6-0bb929dd6194","Type":"ContainerStarted","Data":"b065d6997fcf3b0c98d25c4b77991498dd7e50e960c8d33e96fc1a3c4bbd7f96"} Nov 01 21:43:36.089063 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.089031 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="f7c51de886ef1de1dfeb56c6f4e9eac770f56dcebd8245afd84e6adfd670b35c" exitCode=0 Nov 01 21:43:36.089944 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.089918 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"f7c51de886ef1de1dfeb56c6f4e9eac770f56dcebd8245afd84e6adfd670b35c"} Nov 01 21:43:36.104093 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.104055 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jxl58" podStartSLOduration=2.797304595 podStartE2EDuration="11.104041485s" totalImagesPullingTime="8.30673689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:36.103553336 +0000 UTC m=+11.767835597" watchObservedRunningTime="2026-11-01 21:43:36.104041485 +0000 UTC m=+11.768323728" Nov 01 21:43:36.934151 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.934127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:36.934275 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:36.934250 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:36.934647 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:36.934622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:36.934754 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:36.934740 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:37.093971 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.093937 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" event={"ID":"0b5cff7d-1650-41de-9984-fd5cbd4b3fea","Type":"ContainerStarted","Data":"84ba49d7b31a3740ab01dbd1206d1b1bf939beb6238ff5ce046716cdf2a8fa7c"} Nov 01 21:43:37.099634 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.099601 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="b4735035d86b01d7ac3a6e4171e150e7cc252afc3425445f39c63dbb01508ddf" exitCode=0 Nov 01 21:43:37.099773 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.099748 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"b4735035d86b01d7ac3a6e4171e150e7cc252afc3425445f39c63dbb01508ddf"} Nov 01 21:43:37.103773 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.103749 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9t47" event={"ID":"36c671f8-9973-4f01-a790-c286bb2ee732","Type":"ContainerStarted","Data":"c9d7d458334206a45eac2cd822668abe279e993bdb56622c50b26901244737e6"} Nov 01 21:43:37.121958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.121129 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ckgds" podStartSLOduration=4.282044831 podStartE2EDuration="12.121110403s" totalImagesPullingTime="7.839065572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:36.151246128 +0000 UTC m=+11.815528371" watchObservedRunningTime="2026-11-01 21:43:37.121110403 +0000 UTC m=+12.785392644" Nov 01 21:43:37.121958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.121883 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" podStartSLOduration=3.48508751 podStartE2EDuration="12.121868313s" totalImagesPullingTime="8.636780803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:37.121232845 +0000 UTC m=+12.785515089" watchObservedRunningTime="2026-11-01 21:43:37.121868313 +0000 UTC m=+12.786150610" Nov 01 21:43:37.136839 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.136798 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z9t47" podStartSLOduration=3.939645705 podStartE2EDuration="12.136788429s" totalImagesPullingTime="8.197142724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:37.136289989 +0000 UTC m=+12.800572231" watchObservedRunningTime="2026-11-01 21:43:37.136788429 +0000 UTC m=+12.801070671" Nov 01 21:43:37.934050 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:37.934028 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:37.934154 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:37.934131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:38.109064 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.109031 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="deaea5266bb46a164d4ec3ec180ff50f079398b478f5b2ea66d38fe2cf89072f" exitCode=0 Nov 01 21:43:38.109535 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.109122 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"deaea5266bb46a164d4ec3ec180ff50f079398b478f5b2ea66d38fe2cf89072f"} Nov 01 21:43:38.111486 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.111470 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" event={"ID":"e632f276-6eef-4478-8c2c-1f7e4b98a6d7","Type":"ContainerStarted","Data":"03b7fd184b4b0205d329e4270708f736c4864525411198bd64ab8ac9d7167c2d"} Nov 01 21:43:38.151792 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.151718 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" podStartSLOduration=3.127138027 podStartE2EDuration="13.151702366s" totalImagesPullingTime="10.024564339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:38.15135582 +0000 UTC m=+13.815638064" watchObservedRunningTime="2026-11-01 21:43:38.151702366 +0000 UTC m=+13.815984611" Nov 01 21:43:38.934829 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.934792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:38.935361 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:38.934951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:38.935608 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:38.935366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:38.935608 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:38.935560 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:39.934289 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:39.934057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:39.934847 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:39.934406 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:40.937672 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:40.937640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:40.938115 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:40.937640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:40.938115 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:40.937764 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:40.938115 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:40.937799 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:41.121915 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.121877 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="e4885f2c3a4b1235b8c6f6ec9fa007b087ef88ce89dec89c62815db69fafca05" exitCode=0 Nov 01 21:43:41.122138 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.121932 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"e4885f2c3a4b1235b8c6f6ec9fa007b087ef88ce89dec89c62815db69fafca05"} Nov 01 21:43:41.557110 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.557036 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:41.557259 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.557183 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:41.557790 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.557771 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:43:41.934121 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:41.934059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:41.934251 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:41.934141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:42.126596 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.126566 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"45e0c3d4949a1fde54431f036bbd7992f9a9f7d53670b599c90193cd849ab52c"} Nov 01 21:43:42.130043 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.130021 2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="1ef7fb136498fac797e549fe561c8c95529c4c1b8f7d4a11af718bd396e263dd" exitCode=0 Nov 01 21:43:42.130146 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.130107 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerDied","Data":"1ef7fb136498fac797e549fe561c8c95529c4c1b8f7d4a11af718bd396e263dd"} Nov 01 21:43:42.494953 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.494716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:42.495065 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.494855 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:42.495065 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.495024 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.495008841 +0000 UTC m=+34.159291063 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:42.595855 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.595831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:42.595973 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.595867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:42.596005 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.595978 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:42.596005 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.595997 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:42.596074 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.596012 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:42.596074 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.596024 2578 projected.go:196] Error preparing data for projected volume kube-api-access-r49cn for pod openshift-network-diagnostics/network-check-target-t7lmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:42.596074 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.596041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret podName:bef85c8b-3d71-48ea-95f5-9104415811b2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.596023507 +0000 UTC m=+34.260305740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret") pod "global-pull-secret-syncer-wr8bc" (UID: "bef85c8b-3d71-48ea-95f5-9104415811b2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:42.596074 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.596058 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn podName:a47e4a36-2064-44c4-bf33-1c26e5b7603c nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.596048147 +0000 UTC m=+34.260330389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r49cn" (UniqueName: "kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn") pod "network-check-target-t7lmc" (UID: "a47e4a36-2064-44c4-bf33-1c26e5b7603c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:42.938869 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.938794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:42.938996 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.938893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:42.938996 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:42.938946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:42.939062 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:42.939031 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:43.134743 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.134712 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"1892ba717277e87bc50f77cd959de37c88221f85761ceab0ecadc6a157e355b5"} Nov 01 21:43:43.135212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.134793 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"125287dc3f7fced8752f1a85b8b5d6dace1bfcd42126803c7d060acc904bdd9a"} Nov 01 21:43:43.135212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.134845 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"d2a2d2a65aea8ba23aee190371a7554b9e59ec72a8e4a3a815dab143c67b57f7"} Nov 01 21:43:43.135212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.134892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"b9010d2f22c20e9bb1b51f974068b7020a71d3c8e8eeb54e200d738a38263a27"} Nov 01 21:43:43.135212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.134934 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"424c92e04d37c5bb084c303270c54b6e4a0b84821067b399f0d1ab459a596cce"} Nov 01 21:43:43.137776 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.137751 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4tw88" event={"ID":"3eb0e1e5-b593-4c2b-9950-ae9c967710e1","Type":"ContainerStarted","Data":"c7cb08874cb628dd8f0150ed002f74dbc03a2c13cd8458fe43722784dab7d55e"} Nov 01 21:43:43.163307 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.163272 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4tw88" podStartSLOduration=6.888753735 podStartE2EDuration="18.163261093s" totalImagesPullingTime="11.274507358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:43.161891299 +0000 UTC m=+18.826173542" watchObservedRunningTime="2026-11-01 21:43:43.163261093 +0000 UTC m=+18.827543335" Nov 01 21:43:43.934438 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:43.934404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:43.934593 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:43.934516 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:44.936231 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:44.936201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:44.936975 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:44.936936 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:44.937523 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:44.937493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:44.938003 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:44.937964 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:45.145032 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:45.144997 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"75e2221f9f9fb1d2631a692491e1c7b97531530d66962396ca09571e1044a6a2"} Nov 01 21:43:45.934641 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:45.934612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:45.934807 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:45.934699 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:46.937263 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:46.937013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:46.937263 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:46.937063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:46.937932 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:46.937266 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:46.937932 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:46.937325 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:47.152417 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:47.152382 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" event={"ID":"385fff43-5171-48dd-8530-98bdfb742bec","Type":"ContainerStarted","Data":"42058d9bd843d49c81bd40be818bb3e60bd12896f68ef3f5e872e77087d3bd34"} Nov 01 21:43:47.152755 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:47.152728 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:47.168433 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:47.168413 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:47.186950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:47.186911 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" podStartSLOduration=7.263533512 podStartE2EDuration="22.186895906s" totalImagesPullingTime="14.923362394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:47.186598483 +0000 UTC m=+22.850880725" watchObservedRunningTime="2026-11-01 21:43:47.186895906 +0000 UTC m=+22.851178150" Nov 01 21:43:47.934463 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:47.934413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:47.934643 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:47.934549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:48.155321 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.155296 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:48.155827 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.155381 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:48.170914 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.170891 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:43:48.255189 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.255118 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7lmc"] Nov 01 21:43:48.255336 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.255222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:48.255336 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:48.255323 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:48.258008 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.257984 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wr8bc"] Nov 01 21:43:48.258115 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.258074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:48.258174 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:48.258156 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:48.258579 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.258561 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vlnwn"] Nov 01 21:43:48.258657 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:48.258646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:48.258751 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:48.258733 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:49.933828 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:49.933795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:49.934267 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:49.933869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:49.934267 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:49.933894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:49.934267 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:49.933973 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:49.934267 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:49.934031 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:49.934267 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:49.934104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:51.934736 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:51.934700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:51.935212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:51.934700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:51.935212 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:51.934807 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:51.935212 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:51.934700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:51.935212 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:51.934894 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:51.935212 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:51.934974 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:53.933776 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:53.933741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:53.933776 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:53.933771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:53.934189 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:53.933893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vlnwn" podUID="cfaa13d7-7690-4203-842a-94b26ba0331b" Nov 01 21:43:53.934189 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:53.933932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:53.934189 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:53.933997 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr8bc" podUID="bef85c8b-3d71-48ea-95f5-9104415811b2" Nov 01 21:43:53.934189 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:53.934045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7lmc" podUID="a47e4a36-2064-44c4-bf33-1c26e5b7603c" Nov 01 21:43:54.662170 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.662116 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeReady" Nov 01 21:43:54.662271 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.662232 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 21:43:54.697833 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.697802 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qtm8"] Nov 01 21:43:54.700999 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.700971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:43:54.703780 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.702947 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-chf9m"] Nov 01 21:43:54.707276 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.707257 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfwr9"] Nov 01 21:43:54.707383 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.707366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:43:54.708647 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.708629 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.710797 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.710105 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7"] Nov 01 21:43:54.710797 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.710321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:43:54.713016 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.712999 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-554f8bcb97-6mq5w"] Nov 01 21:43:54.713138 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.713119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.716305 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.716285 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6"] Nov 01 21:43:54.716412 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.716396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.719896 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.719875 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj"] Nov 01 21:43:54.723381 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.723363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.729936 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.729912 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:43:54.730345 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.730319 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.735997 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.735949 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lqtc9"] Nov 01 21:43:54.736149 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.736091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.736895 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.736878 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 21:43:54.739370 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.739349 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 21:43:54.740142 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.740117 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 21:43:54.740142 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.740120 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 21:43:54.740285 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.740128 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9glb6\"" Nov 01 21:43:54.741796 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.741779 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:43:54.741922 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.741906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.744932 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.744916 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6"] Nov 01 21:43:54.745000 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.744990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.747561 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.747543 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jv95p\"" Nov 01 21:43:54.748064 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.748045 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nscn6"] Nov 01 21:43:54.748151 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.748143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:43:54.751016 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.750996 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5fkz5"] Nov 01 21:43:54.751101 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.751059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:54.754152 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.754137 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn"] Nov 01 21:43:54.754247 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.754233 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:54.756960 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.756944 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf"] Nov 01 21:43:54.757065 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.757034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:43:54.762650 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.762629 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qtm8"] Nov 01 21:43:54.762756 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.762669 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-chf9m"] Nov 01 21:43:54.762756 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.762688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:54.762994 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.762693 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfwr9"] Nov 01 21:43:54.763145 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.763124 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-55x22"] Nov 01 21:43:54.766386 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.766367 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wbmm4"] Nov 01 21:43:54.766524 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.766508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55x22" Nov 01 21:43:54.770486 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770469 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7"] Nov 01 21:43:54.770557 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770505 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6"] Nov 01 21:43:54.770557 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:54.770557 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770527 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj"] Nov 01 21:43:54.770557 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770554 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-554f8bcb97-6mq5w"] Nov 01 21:43:54.770709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770576 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5fkz5"] Nov 01 21:43:54.770709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770602 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6"] Nov 01 21:43:54.770709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770625 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:43:54.770709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770644 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lqtc9"] Nov 01 21:43:54.770709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.770664 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn"] Nov 01 21:43:54.781493 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.781466 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbmm4"] Nov 01 21:43:54.794542 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794504 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf"] Nov 01 21:43:54.794634 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.794634 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.794634 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.794634 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.794843 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmccv\" (UniqueName: \"kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.794843 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw67\" (UniqueName: \"kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67\") pod \"kas-connection-checker-685895c7c9-6qtm8\" (UID: \"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8\") " pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:43:54.794940 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2nq\" (UniqueName: \"kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.794940 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csf7t\" (UniqueName: \"kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.795041 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.795041 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.794996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.795137 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.795122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfq5p\" (UniqueName: \"kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p\") pod \"kas-connection-checker-685895c7c9-lfwr9\" (UID: \"cd265d9d-2558-4f1c-8b67-545bf442aa7c\") " pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:43:54.795190 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.795175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.795267 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.795202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.795267 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.795230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.795671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjhd\" (UniqueName: \"kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmxc\" (UniqueName: \"kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc\") pod \"kas-connection-checker-685895c7c9-chf9m\" (UID: \"cef8eaf9-2d56-4d3f-a171-3583becf435e\") " pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.796906 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.796900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.797575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.797087 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:43:54.801174 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.801156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55x22"] Nov 01 21:43:54.802343 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.802245 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nscn6"] Nov 01 21:43:54.807962 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.807942 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.808074 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808062 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.808203 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808176 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.808672 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808619 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.808672 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808656 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.808785 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808694 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 21:43:54.808785 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808654 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.808785 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808704 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.808785 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808732 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.808913 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.808805 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 21:43:54.809164 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809136 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.809241 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809177 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 21:43:54.809241 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809194 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.809241 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809206 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 21:43:54.809241 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809217 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.809401 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809182 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.809401 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.809257 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.810076 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810057 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 21:43:54.810231 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810217 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.810299 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810259 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.810349 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810306 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.810349 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810340 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.810461 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810375 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 21:43:54.810544 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810526 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:54.810598 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810562 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 21:43:54.810598 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.810591 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.811055 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.811042 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 21:43:54.811856 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.811838 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.813700 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.813684 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 21:43:54.816944 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.816921 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 21:43:54.825328 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.825308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 21:43:54.825705 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.825687 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 21:43:54.826076 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.826057 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 21:43:54.828865 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.828846 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 21:43:54.828944 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.828922 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 21:43:54.828996 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.828978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 21:43:54.828996 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.828983 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 21:43:54.829118 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829101 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 21:43:54.829181 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829162 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 21:43:54.829233 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829207 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vwhft\"" Nov 01 21:43:54.829233 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829217 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 21:43:54.829330 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829278 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 21:43:54.829330 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829299 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pn6l\"" Nov 01 21:43:54.829398 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829356 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rcssj\"" Nov 01 21:43:54.829597 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.829577 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 21:43:54.830161 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830145 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fpfrj\"" Nov 01 21:43:54.830214 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830173 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5qkrj\"" Nov 01 21:43:54.830214 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830189 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 21:43:54.830214 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830208 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzsqs\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830214 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bzsh4\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830235 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmv5z\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830246 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n8gq9\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830253 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830267 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hzp42\"" Nov 01 21:43:54.830323 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.830272 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vvt4k\"" Nov 01 21:43:54.897831 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.897807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjhd\" (UniqueName: \"kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.897831 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.897834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmxc\" (UniqueName: \"kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc\") pod \"kas-connection-checker-685895c7c9-chf9m\" (UID: \"cef8eaf9-2d56-4d3f-a171-3583becf435e\") " pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:43:54.897981 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.897857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.897981 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.897875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.897981 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.897941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.898077 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.898126 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.898074 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 21:43:54.898126 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:54.898202 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.898202 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.898157 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls podName:7a4912cf-de1f-4070-9635-47de5fa95ad5 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.398137174 +0000 UTC m=+31.062419414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-bltxj" (UID: "7a4912cf-de1f-4070-9635-47de5fa95ad5") : secret "samples-operator-tls" not found Nov 01 21:43:54.898202 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:54.898304 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:54.898304 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.898304 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.898431 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmccv\" (UniqueName: \"kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.898565 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.898565 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:54.898565 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw67\" (UniqueName: \"kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67\") pod \"kas-connection-checker-685895c7c9-6qtm8\" (UID: \"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8\") " pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:43:54.898565 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfq5p\" (UniqueName: \"kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p\") pod \"kas-connection-checker-685895c7c9-lfwr9\" (UID: \"cd265d9d-2558-4f1c-8b67-545bf442aa7c\") " pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:43:54.898725 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.898725 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:54.898725 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.898845 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2nq\" (UniqueName: \"kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.898845 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.898845 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csf7t\" (UniqueName: \"kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.898958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.898958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.898958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.898958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.898958 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcqf\" (UniqueName: \"kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf\") pod \"network-check-source-6f78d59f99-2k5vn\" (UID: \"fddb723d-930f-4b42-8f63-9780761c0119\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:43:54.899162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.898989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.899162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:54.899162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.899162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.899162 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.899355 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.899403 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.899545 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.899652 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.899635 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:54.899713 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.899655 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:43:54.899762 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.899819 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.899799 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.399783706 +0000 UTC m=+31.064065928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:43:54.899978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.899978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.899978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpx4\" (UniqueName: \"kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.899978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.899978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.899948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxrd\" (UniqueName: \"kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd\") pod \"volume-data-source-validator-58c9f5f46d-q2tv6\" (UID: \"490c53c8-ef5b-4296-80b5-32a3204b483f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:43:54.900236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:54.900236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:54.900236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.900236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:54.900236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth6h\" (UniqueName: \"kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.900326 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.900346 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.400332045 +0000 UTC m=+31.064614278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:54.900379 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.400363173 +0000 UTC m=+31.064645401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnjg\" (UniqueName: \"kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.900506 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpb5\" (UniqueName: \"kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:54.900966 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.900865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwts\" (UniqueName: \"kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:54.901358 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.901338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.902368 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.902340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.902498 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.902472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.902591 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.902518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.902591 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.902577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.902709 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.902694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.903358 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.903341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.912051 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.911998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfq5p\" (UniqueName: \"kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p\") pod \"kas-connection-checker-685895c7c9-lfwr9\" (UID: \"cd265d9d-2558-4f1c-8b67-545bf442aa7c\") " pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:43:54.912206 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.912164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmxc\" (UniqueName: \"kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc\") pod \"kas-connection-checker-685895c7c9-chf9m\" (UID: \"cef8eaf9-2d56-4d3f-a171-3583becf435e\") " pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:43:54.912454 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.912271 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjhd\" (UniqueName: \"kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:54.912454 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.912278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmccv\" (UniqueName: \"kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:54.912454 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.912378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.914865 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.914844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2nq\" (UniqueName: \"kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:54.914973 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.914956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:54.915047 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.915023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csf7t\" (UniqueName: \"kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:54.916014 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:54.915994 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw67\" (UniqueName: \"kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67\") pod \"kas-connection-checker-685895c7c9-6qtm8\" (UID: \"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8\") " pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:43:55.001184 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpx4\" (UniqueName: \"kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxrd\" (UniqueName: \"kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd\") pod \"volume-data-source-validator-58c9f5f46d-q2tv6\" (UID: \"490c53c8-ef5b-4296-80b5-32a3204b483f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth6h\" (UniqueName: \"kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001327 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnjg\" (UniqueName: \"kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001400 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.501382628 +0000 UTC m=+31.165664863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpb5\" (UniqueName: \"kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001482 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001583 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:43:55.001612 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001632 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.501615322 +0000 UTC m=+31.165897571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001666 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.001752 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.50173594 +0000 UTC m=+31.166018165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwts\" (UniqueName: \"kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.001993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.002324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcqf\" (UniqueName: \"kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf\") pod \"network-check-source-6f78d59f99-2k5vn\" (UID: \"fddb723d-930f-4b42-8f63-9780761c0119\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.002677 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.002723 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.502707924 +0000 UTC m=+31.166990172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.003199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.002940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.003899 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.003800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.005309 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.005295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.005978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.005482 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.005978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.005754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.006132 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.006089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:55.006372 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.006331 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.014591 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.014567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwts\" (UniqueName: \"kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:55.015836 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.015815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxrd\" (UniqueName: \"kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd\") pod \"volume-data-source-validator-58c9f5f46d-q2tv6\" (UID: \"490c53c8-ef5b-4296-80b5-32a3204b483f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:43:55.016169 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth6h\" (UniqueName: \"kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.016484 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.016484 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnjg\" (UniqueName: \"kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.016607 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcqf\" (UniqueName: \"kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf\") pod \"network-check-source-6f78d59f99-2k5vn\" (UID: \"fddb723d-930f-4b42-8f63-9780761c0119\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:43:55.016607 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpb5\" (UniqueName: \"kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.016672 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.016625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpx4\" (UniqueName: \"kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.017602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.017586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.019497 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.019476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:43:55.035225 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.035207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:43:55.050550 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.050513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:43:55.056767 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.056744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:43:55.068644 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.068621 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:43:55.091993 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.091970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:43:55.112343 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.108173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:43:55.116126 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.115971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:43:55.132683 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.130263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:43:55.148007 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.147828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:43:55.313183 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.313157 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qtm8"] Nov 01 21:43:55.325382 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.325349 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3665c82e_f3a8_461e_bb7a_d4cb6d6111a8.slice/crio-b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3 WatchSource:0}: Error finding container b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3: Status 404 returned error can't find the container with id b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3 Nov 01 21:43:55.340508 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.338569 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-chf9m"] Nov 01 21:43:55.362327 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.362300 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfwr9"] Nov 01 21:43:55.366270 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.366240 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd265d9d_2558_4f1c_8b67_545bf442aa7c.slice/crio-7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e WatchSource:0}: Error finding container 7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e: Status 404 returned error can't find the container with id 7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e Nov 01 21:43:55.381602 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.381567 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7"] Nov 01 21:43:55.385411 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.385378 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833deb27_8ec7_4e48_9827_a627d0c7fd5c.slice/crio-29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa WatchSource:0}: Error finding container 29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa: Status 404 returned error can't find the container with id 29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa Nov 01 21:43:55.386959 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.386928 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6"] Nov 01 21:43:55.388312 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.388282 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79cdc8f_6b2d_4623_b321_be0088cfad25.slice/crio-305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8 WatchSource:0}: Error finding container 305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8: Status 404 returned error can't find the container with id 305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8 Nov 01 21:43:55.407724 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.407370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.407433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.407708 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.407851 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.407795 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.407774222 +0000 UTC m=+32.072056460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.407923 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.407906371 +0000 UTC m=+32.072188595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.407965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.408069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.408150 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.408199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.408184271 +0000 UTC m=+32.072466509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.408212 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 21:43:55.408286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.408260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls podName:7a4912cf-de1f-4070-9635-47de5fa95ad5 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.408248708 +0000 UTC m=+32.072530954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-bltxj" (UID: "7a4912cf-de1f-4070-9635-47de5fa95ad5") : secret "samples-operator-tls" not found Nov 01 21:43:55.454990 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.454964 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6"] Nov 01 21:43:55.458556 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.458519 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn"] Nov 01 21:43:55.459058 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.459039 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490c53c8_ef5b_4296_80b5_32a3204b483f.slice/crio-5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2 WatchSource:0}: Error finding container 5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2: Status 404 returned error can't find the container with id 5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2 Nov 01 21:43:55.461702 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.461265 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lqtc9"] Nov 01 21:43:55.462016 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.461972 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddb723d_930f_4b42_8f63_9780761c0119.slice/crio-d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda WatchSource:0}: Error finding container d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda: Status 404 returned error can't find the container with id d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda Nov 01 21:43:55.463868 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.463787 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3b2036_627e_4704_91fb_87df2b35c328.slice/crio-08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7 WatchSource:0}: Error finding container 08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7: Status 404 returned error can't find the container with id 08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7 Nov 01 21:43:55.508733 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.508711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:55.508854 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.508747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:55.508854 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.508780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508867 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508874 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508893 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508920 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.508905649 +0000 UTC m=+32.173187881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508867 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:43:55.508970 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508950 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.508937262 +0000 UTC m=+32.173219485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:43:55.509266 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.508975 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.508964643 +0000 UTC m=+32.173246864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:43:55.509266 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.509073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:55.509266 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.509202 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:55.509266 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:55.509257 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:43:56.509240645 +0000 UTC m=+32.173522866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:55.525130 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.525028 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbmm4"] Nov 01 21:43:55.527879 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.527830 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nscn6"] Nov 01 21:43:55.527879 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.527839 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cb1d59_4690_49c0_95d5_ecf9ce11ff7d.slice/crio-f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43 WatchSource:0}: Error finding container f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43: Status 404 returned error can't find the container with id f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43 Nov 01 21:43:55.529930 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:55.529912 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510010af_f8ad_4b48_9f73_e9ad1b0a2de9.slice/crio-80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63 WatchSource:0}: Error finding container 80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63: Status 404 returned error can't find the container with id 80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63 Nov 01 21:43:55.934605 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.934566 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:55.935040 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.935024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:55.935120 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.935078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:55.940032 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.940009 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 21:43:55.940548 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.940530 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\"" Nov 01 21:43:55.940548 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.940540 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\"" Nov 01 21:43:55.940683 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.940608 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\"" Nov 01 21:43:55.940683 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:55.940646 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 21:43:56.181739 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.181698 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" event={"ID":"fddb723d-930f-4b42-8f63-9780761c0119","Type":"ContainerStarted","Data":"d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda"} Nov 01 21:43:56.183791 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.183757 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" event={"ID":"510010af-f8ad-4b48-9f73-e9ad1b0a2de9","Type":"ContainerStarted","Data":"80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63"} Nov 01 21:43:56.187740 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.187671 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" event={"ID":"cd265d9d-2558-4f1c-8b67-545bf442aa7c","Type":"ContainerStarted","Data":"e0f9b9005a7023e8894a781284e2c03e3a0013964154f5910c26c2b9e49d2eed"} Nov 01 21:43:56.187740 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.187722 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" event={"ID":"cd265d9d-2558-4f1c-8b67-545bf442aa7c","Type":"ContainerStarted","Data":"7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e"} Nov 01 21:43:56.190119 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.190093 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbmm4" event={"ID":"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d","Type":"ContainerStarted","Data":"f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43"} Nov 01 21:43:56.192462 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.192429 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" event={"ID":"a79cdc8f-6b2d-4623-b321-be0088cfad25","Type":"ContainerStarted","Data":"305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8"} Nov 01 21:43:56.193859 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.193826 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" event={"ID":"833deb27-8ec7-4e48-9827-a627d0c7fd5c","Type":"ContainerStarted","Data":"29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa"} Nov 01 21:43:56.196077 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.196055 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-chf9m" event={"ID":"cef8eaf9-2d56-4d3f-a171-3583becf435e","Type":"ContainerStarted","Data":"2d049c54abcc21b9139cdea59e0eeb8063349450974c4d6cacded226b2fd5439"} Nov 01 21:43:56.196174 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.196093 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-chf9m" event={"ID":"cef8eaf9-2d56-4d3f-a171-3583becf435e","Type":"ContainerStarted","Data":"49d4013610c4835099eb881e206ba41b36b02b91d216764e9c59d6738c7ffd57"} Nov 01 21:43:56.198615 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.198384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" event={"ID":"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8","Type":"ContainerStarted","Data":"c72c7b799506729de8e602107556b77cfa7e54652e47dbae8caf9160e377c4a0"} Nov 01 21:43:56.198615 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.198431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" event={"ID":"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8","Type":"ContainerStarted","Data":"b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3"} Nov 01 21:43:56.200683 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.200639 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" event={"ID":"490c53c8-ef5b-4296-80b5-32a3204b483f","Type":"ContainerStarted","Data":"5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2"} Nov 01 21:43:56.202105 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.202080 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" event={"ID":"2f3b2036-627e-4704-91fb-87df2b35c328","Type":"ContainerStarted","Data":"08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7"} Nov 01 21:43:56.211855 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.211805 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" podStartSLOduration=411.211789213 podStartE2EDuration="6m51.211789213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:56.2113866 +0000 UTC m=+31.875668843" watchObservedRunningTime="2026-11-01 21:43:56.211789213 +0000 UTC m=+31.876071458" Nov 01 21:43:56.237668 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.237606 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:56.255250 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.255208 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-chf9m" podStartSLOduration=411.255196717 podStartE2EDuration="6m51.255196717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:56.254981267 +0000 UTC m=+31.919263511" watchObservedRunningTime="2026-11-01 21:43:56.255196717 +0000 UTC m=+31.919478961" Nov 01 21:43:56.255746 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.255709 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" podStartSLOduration=411.255702236 podStartE2EDuration="6m51.255702236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:56.235318749 +0000 UTC m=+31.899600992" watchObservedRunningTime="2026-11-01 21:43:56.255702236 +0000 UTC m=+31.919984481" Nov 01 21:43:56.402418 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.402325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.417131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.417243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.417294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.417331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.417519 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.417578 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.417561515 +0000 UTC m=+34.081843750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418003 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418051 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls podName:7a4912cf-de1f-4070-9635-47de5fa95ad5 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.418036754 +0000 UTC m=+34.082318988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-bltxj" (UID: "7a4912cf-de1f-4070-9635-47de5fa95ad5") : secret "samples-operator-tls" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418112 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418123 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.418145337 +0000 UTC m=+34.082427561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:43:56.418249 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.418216 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.418204143 +0000 UTC m=+34.082486380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.518086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.518182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.518227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.518270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518475 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518553 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.51853369 +0000 UTC m=+34.182815929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518651 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:56.518703 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518697 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:56.519286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518720 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:43:56.519286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518701 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.518687191 +0000 UTC m=+34.182969416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:56.519286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518762 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:43:56.519286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518787 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.518769557 +0000 UTC m=+34.183051784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:43:56.519286 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:56.518818 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:58.518803547 +0000 UTC m=+34.183085774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:43:56.576336 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:56.576247 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:57.851324 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:57.851243 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:58.438826 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.438789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:43:58.439014 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.438857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:58.439014 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.438899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:43:58.439014 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.438992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:43:58.439181 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439167 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 21:43:58.439237 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439228 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls podName:7a4912cf-de1f-4070-9635-47de5fa95ad5 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.439210012 +0000 UTC m=+38.103492247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-bltxj" (UID: "7a4912cf-de1f-4070-9635-47de5fa95ad5") : secret "samples-operator-tls" not found Nov 01 21:43:58.439636 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439621 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:58.439687 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439641 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:43:58.439687 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.439669728 +0000 UTC m=+38.103951966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:43:58.439788 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439763 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.439751837 +0000 UTC m=+38.104034071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:43:58.439844 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439823 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:43:58.439893 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.439855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.439844293 +0000 UTC m=+38.104126528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:43:58.540194 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.540164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:43:58.540372 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.540225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:43:58.540372 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.540250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:43:58.540372 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.540275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:43:58.540372 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.540295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:43:58.540609 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540408 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 21:43:58.540609 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540481 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs podName:cfaa13d7-7690-4203-842a-94b26ba0331b nodeName:}" failed. No retries permitted until 2026-11-01 21:44:30.540462497 +0000 UTC m=+66.204744733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs") pod "network-metrics-daemon-vlnwn" (UID: "cfaa13d7-7690-4203-842a-94b26ba0331b") : secret "metrics-daemon-secret" not found Nov 01 21:43:58.540609 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540549 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:58.540609 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540580 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.540570123 +0000 UTC m=+38.204852344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:43:58.540816 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540637 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:43:58.540816 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540669 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.540658872 +0000 UTC m=+38.204941098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:43:58.540816 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540726 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:43:58.540816 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540737 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:43:58.540816 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540770 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.540760121 +0000 UTC m=+38.205042355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:43:58.541061 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540827 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:43:58.541061 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:43:58.540856 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.540845857 +0000 UTC m=+38.205128079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:43:58.641057 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.641017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:58.641225 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.641160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:58.649810 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.649782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:58.658389 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.658341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:58.668019 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.667993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:43:58.696882 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.696853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:43:58.988546 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:58.987604 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7lmc"] Nov 01 21:43:58.992375 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:58.992337 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47e4a36_2064_44c4_bf33_1c26e5b7603c.slice/crio-62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035 WatchSource:0}: Error finding container 62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035: Status 404 returned error can't find the container with id 62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035 Nov 01 21:43:59.026620 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.026593 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wr8bc"] Nov 01 21:43:59.034729 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:43:59.034682 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef85c8b_3d71_48ea_95f5_9104415811b2.slice/crio-ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c WatchSource:0}: Error finding container ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c: Status 404 returned error can't find the container with id ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c Nov 01 21:43:59.229275 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.229196 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wr8bc" event={"ID":"bef85c8b-3d71-48ea-95f5-9104415811b2","Type":"ContainerStarted","Data":"ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c"} Nov 01 21:43:59.231016 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.230987 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7lmc" event={"ID":"a47e4a36-2064-44c4-bf33-1c26e5b7603c","Type":"ContainerStarted","Data":"62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035"} Nov 01 21:43:59.233847 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.233154 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" event={"ID":"490c53c8-ef5b-4296-80b5-32a3204b483f","Type":"ContainerStarted","Data":"bbefc01813aa7ef968e270c939e53d3e660e9eb676e5138025f4ee83f1208239"} Nov 01 21:43:59.253896 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.253855 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" podStartSLOduration=381.48765451 podStartE2EDuration="6m24.253842685s" totalImagesPullingTime="2.766188175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:59.252480026 +0000 UTC m=+34.916762269" watchObservedRunningTime="2026-11-01 21:43:59.253842685 +0000 UTC m=+34.918124932" Nov 01 21:43:59.393471 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.393349 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:59.516460 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:43:59.516368 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:01.244999 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.244923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/0.log" Nov 01 21:44:01.245424 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.245008 2578 generic.go:356] "Generic (PLEG): container finished" podID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" containerID="4f3ceb564ff64c49b8eef957dca156397ce5140a6f09de9245650d79c33a96f1" exitCode=255 Nov 01 21:44:01.245424 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.245104 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" event={"ID":"510010af-f8ad-4b48-9f73-e9ad1b0a2de9","Type":"ContainerDied","Data":"4f3ceb564ff64c49b8eef957dca156397ce5140a6f09de9245650d79c33a96f1"} Nov 01 21:44:01.245424 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.245282 2578 scope.go:122] "RemoveContainer" containerID="4f3ceb564ff64c49b8eef957dca156397ce5140a6f09de9245650d79c33a96f1" Nov 01 21:44:01.249684 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.249653 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" event={"ID":"a79cdc8f-6b2d-4623-b321-be0088cfad25","Type":"ContainerStarted","Data":"5916ddec60564c7ef70909f124c69a68c93cc376218e34ef94f5c4f84609526c"} Nov 01 21:44:01.252170 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.252144 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" event={"ID":"833deb27-8ec7-4e48-9827-a627d0c7fd5c","Type":"ContainerStarted","Data":"7117a20130c7d82b73be1034a63b4cc02bf3e9f4d1e8dbb94ef354bf9ff0cbe8"} Nov 01 21:44:01.254305 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.254283 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" event={"ID":"2f3b2036-627e-4704-91fb-87df2b35c328","Type":"ContainerStarted","Data":"ecbe335295609b5460bd8e51923d0ad961599a1b51cf4917ee4aefd4382142ca"} Nov 01 21:44:01.290882 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.290736 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" podStartSLOduration=384.201045002 podStartE2EDuration="6m29.29071757s" totalImagesPullingTime="5.089672568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:01.288878694 +0000 UTC m=+36.953160938" watchObservedRunningTime="2026-11-01 21:44:01.29071757 +0000 UTC m=+36.954999832" Nov 01 21:44:01.336085 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.336030 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" podStartSLOduration=383.804262224 podStartE2EDuration="6m29.336012421s" totalImagesPullingTime="5.531750197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:01.316413993 +0000 UTC m=+36.980696235" watchObservedRunningTime="2026-11-01 21:44:01.336012421 +0000 UTC m=+37.000294665" Nov 01 21:44:01.336553 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.336534 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn"] Nov 01 21:44:01.341685 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.341643 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" podStartSLOduration=383.319555161 podStartE2EDuration="6m28.341629608s" totalImagesPullingTime="5.022074447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:01.341587175 +0000 UTC m=+37.005869419" watchObservedRunningTime="2026-11-01 21:44:01.341629608 +0000 UTC m=+37.005911850" Nov 01 21:44:01.354922 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.354901 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws"] Nov 01 21:44:01.361618 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.355161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:01.363271 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.356781 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:01.365896 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.365859 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 21:44:01.366171 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.366147 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-46fjd\"" Nov 01 21:44:01.387714 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.387155 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn"] Nov 01 21:44:01.387714 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.387197 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws"] Nov 01 21:44:01.387714 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.387281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:01.440026 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.440000 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4bjns"] Nov 01 21:44:01.461896 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.461871 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:44:01.462063 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.462007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.465610 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.465589 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:01.465891 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.465844 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 21:44:01.465891 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.465861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 21:44:01.466045 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.465898 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pth5p\"" Nov 01 21:44:01.472136 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.472116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bz8g\" (UniqueName: \"kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g\") pod \"migrator-66df5d496b-zhwws\" (UID: \"d9ae9063-f7f1-450b-adbf-b21c148dea72\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:01.472231 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.472216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgk8\" (UniqueName: \"kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8\") pod \"migrator-66df5d496b-66qrn\" (UID: \"91699fd6-b0b6-45c7-8d3c-5175c3a30979\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:01.481891 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.481864 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:01.505809 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.505343 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bjns"] Nov 01 21:44:01.505809 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.505418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.505809 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.505493 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:44:01.573199 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.573313 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7vn\" (UniqueName: \"kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.573313 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.573410 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.573410 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgk8\" (UniqueName: \"kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8\") pod \"migrator-66df5d496b-66qrn\" (UID: \"91699fd6-b0b6-45c7-8d3c-5175c3a30979\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:01.573410 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.573585 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.573563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bz8g\" (UniqueName: \"kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g\") pod \"migrator-66df5d496b-zhwws\" (UID: \"d9ae9063-f7f1-450b-adbf-b21c148dea72\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:01.588766 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.588739 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bz8g\" (UniqueName: \"kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g\") pod \"migrator-66df5d496b-zhwws\" (UID: \"d9ae9063-f7f1-450b-adbf-b21c148dea72\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:01.588999 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.588980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgk8\" (UniqueName: \"kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8\") pod \"migrator-66df5d496b-66qrn\" (UID: \"91699fd6-b0b6-45c7-8d3c-5175c3a30979\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:01.620674 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.620653 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.674809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.674854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:01.674921 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.674936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.674961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:01.674989 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.174970983 +0000 UTC m=+37.839253205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : configmap "kube-rbac-proxy" not found Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7vn\" (UniqueName: \"kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:01.675350 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:01.675383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:02.17537272 +0000 UTC m=+37.839654942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:01.675575 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.675547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.687880 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.687817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7vn\" (UniqueName: \"kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:01.697014 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.696877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:01.724645 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.724623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:01.777209 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.777179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.777337 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.777245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.777337 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.777305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.777530 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.777413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.778067 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.778042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.778812 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.778786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.779376 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.779352 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.810867 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.808924 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:01.856078 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:01.854946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:02.019651 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.019620 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn"] Nov 01 21:44:02.027792 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:44:02.025545 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91699fd6_b0b6_45c7_8d3c_5175c3a30979.slice/crio-966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3 WatchSource:0}: Error finding container 966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3: Status 404 returned error can't find the container with id 966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3 Nov 01 21:44:02.084461 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.084415 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws"] Nov 01 21:44:02.089680 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:44:02.089653 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ae9063_f7f1_450b_adbf_b21c148dea72.slice/crio-8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0 WatchSource:0}: Error finding container 8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0: Status 404 returned error can't find the container with id 8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0 Nov 01 21:44:02.136693 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.136656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:02.166997 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.166972 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:44:02.169700 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:44:02.169669 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793bff98_0107_4731_a6a0_5747f9303443.slice/crio-307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516 WatchSource:0}: Error finding container 307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516: Status 404 returned error can't find the container with id 307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516 Nov 01 21:44:02.183345 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.183317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:02.183474 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.183410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:02.183530 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.183468 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:02.183653 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.183534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:03.183513271 +0000 UTC m=+38.847795513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:02.183653 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.183568 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 21:44:02.183653 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.183616 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:03.183601292 +0000 UTC m=+38.847883532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : configmap "kube-rbac-proxy" not found Nov 01 21:44:02.259202 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.259164 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:02.260707 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.260681 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbmm4" event={"ID":"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d","Type":"ContainerStarted","Data":"e1ea0d06879a0509f881582999afdbe317b976f7512a29bd7017ca0f294f2e1a"} Nov 01 21:44:02.263851 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.263826 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" event={"ID":"793bff98-0107-4731-a6a0-5747f9303443","Type":"ContainerStarted","Data":"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e"} Nov 01 21:44:02.263930 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.263872 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" event={"ID":"793bff98-0107-4731-a6a0-5747f9303443","Type":"ContainerStarted","Data":"307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516"} Nov 01 21:44:02.265302 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.265279 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" event={"ID":"91699fd6-b0b6-45c7-8d3c-5175c3a30979","Type":"ContainerStarted","Data":"966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3"} Nov 01 21:44:02.266523 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.266499 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" event={"ID":"d9ae9063-f7f1-450b-adbf-b21c148dea72","Type":"ContainerStarted","Data":"8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0"} Nov 01 21:44:02.268978 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.268568 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" event={"ID":"fddb723d-930f-4b42-8f63-9780761c0119","Type":"ContainerStarted","Data":"e32a894194973f11899e4c7c1e380b4808c5f887e3e4e417e933eccebac82fbd"} Nov 01 21:44:02.270903 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.270861 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.271333 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/0.log" Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.271378 2578 generic.go:356] "Generic (PLEG): container finished" podID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" exitCode=255 Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.271491 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" event={"ID":"510010af-f8ad-4b48-9f73-e9ad1b0a2de9","Type":"ContainerDied","Data":"b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a"} Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.271550 2578 scope.go:122] "RemoveContainer" containerID="4f3ceb564ff64c49b8eef957dca156397ce5140a6f09de9245650d79c33a96f1" Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.271644 2578 scope.go:122] "RemoveContainer" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" Nov 01 21:44:02.271950 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.271830 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nscn6_openshift-console-operator(510010af-f8ad-4b48-9f73-e9ad1b0a2de9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" podUID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" Nov 01 21:44:02.282892 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.282845 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wbmm4" podStartSLOduration=2.214911673 podStartE2EDuration="8.282828819s" totalImagesPullingTime="6.067917146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:02.281691459 +0000 UTC m=+37.945973704" watchObservedRunningTime="2026-11-01 21:44:02.282828819 +0000 UTC m=+37.947111066" Nov 01 21:44:02.287002 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.286970 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6w2xf"] Nov 01 21:44:02.300820 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.300784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.307333 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.307300 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 21:44:02.308597 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.308527 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fzwwl\"" Nov 01 21:44:02.309667 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.308819 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 21:44:02.309667 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.308869 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 21:44:02.309667 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.308886 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 21:44:02.310253 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.310217 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6w2xf"] Nov 01 21:44:02.311369 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.310772 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" podStartSLOduration=378.162195086 podStartE2EDuration="6m24.310755883s" totalImagesPullingTime="6.148560797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:02.304726915 +0000 UTC m=+37.969009160" watchObservedRunningTime="2026-11-01 21:44:02.310755883 +0000 UTC m=+37.975038126" Nov 01 21:44:02.347659 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.347610 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" podStartSLOduration=1.347591551 podStartE2EDuration="1.347591551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:02.345580284 +0000 UTC m=+38.009862528" watchObservedRunningTime="2026-11-01 21:44:02.347591551 +0000 UTC m=+38.011873795" Nov 01 21:44:02.384985 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.384957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.385147 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.385122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsn4\" (UniqueName: \"kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.385539 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.385514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.486082 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.486159 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:02.486159 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:02.486235 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.486290 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:02.486329 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsn4\" (UniqueName: \"kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.486403 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486571 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486595 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486647 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.48662911 +0000 UTC m=+46.150911345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486725 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.486711512 +0000 UTC m=+46.150993749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486788 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486824 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.486811815 +0000 UTC m=+46.151094056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.486831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486926 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 21:44:02.487176 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.486977 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls podName:7a4912cf-de1f-4070-9635-47de5fa95ad5 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.486959646 +0000 UTC m=+46.151241869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-bltxj" (UID: "7a4912cf-de1f-4070-9635-47de5fa95ad5") : secret "samples-operator-tls" not found Nov 01 21:44:02.489412 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.489403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.499197 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.499175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsn4\" (UniqueName: \"kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.587227 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.587196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:02.587360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.587243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:02.587360 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.587328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:02.587360 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587352 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:02.587570 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587365 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:44:02.587570 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587467 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:44:02.587570 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587502 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.587401777 +0000 UTC m=+46.251684022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:44:02.587570 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587503 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:44:02.587727 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.587582965 +0000 UTC m=+46.251865191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:44:02.587727 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.587619888 +0000 UTC m=+46.251902112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:44:02.587852 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.587726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:02.587903 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587888 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:02.587960 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:02.587935 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:44:10.587921564 +0000 UTC m=+46.252203859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:02.635302 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.635270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:02.727949 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.727922 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:02.859270 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.859189 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:02.928551 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:02.928478 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6w2xf"] Nov 01 21:44:02.932229 ip-10-0-131-110 kubenswrapper[2578]: W1101 21:44:02.932200 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0693fb2b_eba1_4b24_a44b_1a98ddfedd54.slice/crio-4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce WatchSource:0}: Error finding container 4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce: Status 404 returned error can't find the container with id 4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce Nov 01 21:44:03.197246 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.197157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:03.197406 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.197307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:03.197501 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:03.197466 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:03.197589 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:03.197572 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:05.197518628 +0000 UTC m=+40.861800867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:03.197937 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:03.197923 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 21:44:03.197975 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:03.197966 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:05.19795337 +0000 UTC m=+40.862235604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : configmap "kube-rbac-proxy" not found Nov 01 21:44:03.277124 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.277096 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:03.277624 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.277606 2578 scope.go:122] "RemoveContainer" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" Nov 01 21:44:03.277893 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:03.277861 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nscn6_openshift-console-operator(510010af-f8ad-4b48-9f73-e9ad1b0a2de9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" podUID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" Nov 01 21:44:03.278948 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.278924 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" event={"ID":"0693fb2b-eba1-4b24-a44b-1a98ddfedd54","Type":"ContainerStarted","Data":"e8223be3131bcbaf31f7ba1322c0cabb8813292b5c2eaabd947ede22840a1d28"} Nov 01 21:44:03.279060 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.278997 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" event={"ID":"0693fb2b-eba1-4b24-a44b-1a98ddfedd54","Type":"ContainerStarted","Data":"4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce"} Nov 01 21:44:03.280578 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.280544 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7lmc" event={"ID":"a47e4a36-2064-44c4-bf33-1c26e5b7603c","Type":"ContainerStarted","Data":"0765a1de5917cef79e01f1e3a0339e6de7b049c84b12746da153905b8d44b15d"} Nov 01 21:44:03.280949 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.280922 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:44:03.317374 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.315836 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" podStartSLOduration=1.315819453 podStartE2EDuration="1.315819453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:03.315264466 +0000 UTC m=+38.979546714" watchObservedRunningTime="2026-11-01 21:44:03.315819453 +0000 UTC m=+38.980101696" Nov 01 21:44:03.335168 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.334280 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7lmc" podStartSLOduration=34.844905822 podStartE2EDuration="38.334267297s" totalImagesPullingTime="3.489361475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:03.332897319 +0000 UTC m=+38.997179575" watchObservedRunningTime="2026-11-01 21:44:03.334267297 +0000 UTC m=+38.998549540" Nov 01 21:44:03.497332 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.497258 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:03.870924 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:03.870884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:04.070204 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:04.070170 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:04.223456 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:04.223415 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:04.354431 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:04.354400 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:04.487209 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:04.487134 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:04.573646 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:04.573618 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jxl58_89d6f94c-7d2b-420b-bea6-0bb929dd6194/dns-node-resolver/0.log" Nov 01 21:44:05.117080 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.117046 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:05.117236 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.117109 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:05.117484 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.117463 2578 scope.go:122] "RemoveContainer" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" Nov 01 21:44:05.117686 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:05.117662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nscn6_openshift-console-operator(510010af-f8ad-4b48-9f73-e9ad1b0a2de9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" podUID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" Nov 01 21:44:05.222243 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.222206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:05.222412 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:05.222338 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 21:44:05.222412 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:05.222408 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.222388232 +0000 UTC m=+44.886670472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : configmap "kube-rbac-proxy" not found Nov 01 21:44:05.222632 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.222605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:05.222800 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:05.222777 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:05.222905 ip-10-0-131-110 kubenswrapper[2578]: E1101 21:44:05.222828 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.222815676 +0000 UTC m=+44.887097902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:05.289290 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.289255 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" event={"ID":"91699fd6-b0b6-45c7-8d3c-5175c3a30979","Type":"ContainerStarted","Data":"3da2922d72f99cf38b00118b8a5441340adcf1483d7d74c0709fe04154265113"} Nov 01 21:44:05.291345 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.291289 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" event={"ID":"d9ae9063-f7f1-450b-adbf-b21c148dea72","Type":"ContainerStarted","Data":"79005f9baae695f06791b50d7fb5c63d67e3db2ef6bea4c0dc3446f8c5b10893"} Nov 01 21:44:05.307342 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.307296 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" podStartSLOduration=2.128735618 podStartE2EDuration="4.307284707s" totalImagesPullingTime="2.178549089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:05.30673103 +0000 UTC m=+40.971013277" watchObservedRunningTime="2026-11-01 21:44:05.307284707 +0000 UTC m=+40.971566963" Nov 01 21:44:05.482056 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.481988 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:05.689349 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.689296 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:05.772353 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.772333 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2cpx_8e6c0a65-773b-45c9-b18b-bd582d7a6307/node-ca/0.log" Nov 01 21:44:05.896599 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:05.896572 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:06.034956 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:06.034932 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:06.153647 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:06.153614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:06.259468 ip-10-0-131-110 kubenswrapper[2578]: I1101 21:44:06.258121 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:44:06.258321 ip-10-0-131-110 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 21:44:06.266818 ip-10-0-131-110 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 21:44:06.266984 ip-10-0-131-110 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 21:44:06.267030 ip-10-0-131-110 systemd[1]: kubelet.service: Consumed 3.153s CPU time. Nov 01 21:44:06.277033 ip-10-0-131-110 systemd[1]: Starting Kubernetes Kubelet... Nov 01 21:44:06.352226 ip-10-0-131-110 kubenswrapper[5036]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:06.352226 ip-10-0-131-110 kubenswrapper[5036]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 21:44:06.352226 ip-10-0-131-110 kubenswrapper[5036]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:06.352226 ip-10-0-131-110 kubenswrapper[5036]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:06.354381 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354308 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.354381 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354383 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354389 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354405 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354409 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354412 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354416 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354419 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354421 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354424 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354427 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354429 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354432 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354435 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354437 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354440 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354453 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354463 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354466 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354469 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354471 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.354472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354474 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354477 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354480 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354483 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354486 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354489 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354492 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354495 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354497 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354500 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354503 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354505 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354508 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354510 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354512 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354515 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354517 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354520 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354522 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354525 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.354950 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354527 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354530 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354534 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354538 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354541 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354545 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354548 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354551 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354554 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354556 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354559 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354562 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354564 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354567 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354569 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354572 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354575 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354577 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354580 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.355469 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354582 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354585 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354588 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354590 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354593 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354595 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354600 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354604 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354607 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354610 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354613 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354616 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354619 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354621 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354624 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354627 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354630 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354633 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354639 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.355934 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354642 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354645 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354647 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354650 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354653 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354655 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354658 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354660 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354663 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354666 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354668 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354670 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354673 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354675 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354678 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354681 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354684 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354686 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354689 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.356390 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354691 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354694 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354697 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354699 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354706 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354711 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354713 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.354716 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355665 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355675 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355678 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355681 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355684 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355687 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355691 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355694 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355696 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355699 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355702 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355705 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.356899 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355707 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355710 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355713 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355716 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355718 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355721 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355723 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355726 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355728 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355731 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355734 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355737 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355739 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355742 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355744 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355747 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355752 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355755 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355757 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355761 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.357371 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355764 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355766 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355769 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355771 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355774 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355777 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355780 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355783 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355786 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355788 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355791 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355794 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355796 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355799 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355801 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355804 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355809 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355813 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355817 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.357905 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355819 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355822 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355825 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355828 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355830 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355833 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355835 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355838 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355841 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355845 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355848 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355850 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355853 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355856 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355858 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355861 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355864 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355866 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355869 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355871 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.358409 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355874 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355877 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355880 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355882 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355885 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355888 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355891 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355893 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355896 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355899 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355902 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355904 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355907 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355909 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355911 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355914 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355917 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355919 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355922 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.358929 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355924 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355927 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355931 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355933 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355936 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355939 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355942 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355947 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355950 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355952 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355955 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355958 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355960 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355963 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355967 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.355970 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356516 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356526 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356529 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.359503 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356532 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356535 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356537 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356540 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356543 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356545 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356548 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356551 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356554 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356557 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356561 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356566 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356582 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356585 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356588 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356593 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356595 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356598 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356600 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356603 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.359965 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356606 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356608 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356613 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356616 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356618 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356621 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356623 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356627 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356630 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356632 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356635 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356638 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356640 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356643 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356645 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356648 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356652 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356655 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356659 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.360500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356661 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356664 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356667 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356669 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356672 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356674 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356676 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356679 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356683 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356685 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356688 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356691 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356694 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356696 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356698 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356703 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356706 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356709 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356712 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356714 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.360964 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356717 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356720 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356723 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356725 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356728 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356742 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356745 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356749 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356752 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356755 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356757 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356760 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356763 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356765 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356768 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356770 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356773 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356775 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356778 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356781 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.361549 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356784 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356787 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356790 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356792 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356795 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356797 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356800 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356805 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356809 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356812 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356815 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356817 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356820 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356822 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356825 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356828 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356832 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356834 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356837 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.362033 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356840 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356842 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356844 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356847 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.356850 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356938 5036 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356946 5036 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356951 5036 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356956 5036 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356961 5036 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356964 5036 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356968 5036 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356972 5036 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356977 5036 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356980 5036 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356984 5036 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356987 5036 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356990 5036 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356993 5036 flags.go:64] FLAG: --cgroup-root="" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.356996 5036 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357001 5036 flags.go:64] FLAG: --client-ca-file="" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357004 5036 flags.go:64] FLAG: --cloud-provider="external" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357007 5036 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 21:44:06.362500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357012 5036 flags.go:64] FLAG: --cluster-domain="" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357015 5036 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357018 5036 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357021 5036 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357024 5036 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357030 5036 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357033 5036 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357036 5036 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357040 5036 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357043 5036 flags.go:64] FLAG: --contention-profiling="false" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357046 5036 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357049 5036 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357052 5036 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357055 5036 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357060 5036 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357063 5036 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357066 5036 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357069 5036 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357072 5036 flags.go:64] FLAG: --enable-server="true" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357075 5036 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357079 5036 flags.go:64] FLAG: --event-burst="100" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357082 5036 flags.go:64] FLAG: --event-qps="50" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357087 5036 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357090 5036 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 21:44:06.363045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357094 5036 flags.go:64] FLAG: --eviction-hard="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357098 5036 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357101 5036 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357104 5036 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357107 5036 flags.go:64] FLAG: --eviction-soft="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357112 5036 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357115 5036 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357118 5036 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357122 5036 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357125 5036 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357128 5036 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357131 5036 flags.go:64] FLAG: --feature-gates="" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357135 5036 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357139 5036 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357142 5036 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357146 5036 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357149 5036 flags.go:64] FLAG: --healthz-port="10248" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357152 5036 flags.go:64] FLAG: --help="false" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357155 5036 flags.go:64] FLAG: --hostname-override="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357158 5036 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357161 5036 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357164 5036 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357168 5036 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 21:44:06.363636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357171 5036 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357174 5036 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357177 5036 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357180 5036 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357183 5036 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357186 5036 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357189 5036 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357194 5036 flags.go:64] FLAG: --kube-reserved="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357197 5036 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357200 5036 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357203 5036 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357206 5036 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357209 5036 flags.go:64] FLAG: --lock-file="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357212 5036 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357216 5036 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357219 5036 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357225 5036 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357228 5036 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357231 5036 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357234 5036 flags.go:64] FLAG: --logging-format="text" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357237 5036 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357240 5036 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357243 5036 flags.go:64] FLAG: --manifest-url="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357246 5036 flags.go:64] FLAG: --manifest-url-header="" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357251 5036 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 21:44:06.364225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357254 5036 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357258 5036 flags.go:64] FLAG: --max-pods="110" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357262 5036 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357265 5036 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357268 5036 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357271 5036 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357274 5036 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357277 5036 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357280 5036 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357286 5036 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357290 5036 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357293 5036 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357296 5036 flags.go:64] FLAG: --pod-cidr="" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357299 5036 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357304 5036 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357307 5036 flags.go:64] FLAG: --pods-per-core="0" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357310 5036 flags.go:64] FLAG: --port="10250" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357313 5036 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357316 5036 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09bd62f6425b1bfb0" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357320 5036 flags.go:64] FLAG: --qos-reserved="" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357323 5036 flags.go:64] FLAG: --read-only-port="10255" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357327 5036 flags.go:64] FLAG: --register-node="true" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357330 5036 flags.go:64] FLAG: --register-with-taints="" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357334 5036 flags.go:64] FLAG: --registry-burst="10" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357337 5036 flags.go:64] FLAG: --registry-qps="5" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357342 5036 flags.go:64] FLAG: --reserved-cpus="" Nov 01 21:44:06.364911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357347 5036 flags.go:64] FLAG: --reserved-memory="" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357353 5036 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357359 5036 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357364 5036 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357369 5036 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357374 5036 flags.go:64] FLAG: --runonce="false" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357378 5036 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357382 5036 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357385 5036 flags.go:64] FLAG: --seccomp-default="false" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357388 5036 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357391 5036 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357394 5036 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357397 5036 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357400 5036 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357403 5036 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357407 5036 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357410 5036 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357413 5036 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357416 5036 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357419 5036 flags.go:64] FLAG: --system-cgroups="" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357423 5036 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357428 5036 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357431 5036 flags.go:64] FLAG: --tls-cert-file="" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357437 5036 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357457 5036 flags.go:64] FLAG: --tls-min-version="" Nov 01 21:44:06.365875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357462 5036 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357466 5036 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357469 5036 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357472 5036 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357476 5036 flags.go:64] FLAG: --v="2" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357488 5036 flags.go:64] FLAG: --version="false" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357492 5036 flags.go:64] FLAG: --vmodule="" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357496 5036 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.357499 5036 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357643 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357648 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357652 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357655 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357657 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357661 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357663 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357666 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357669 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357672 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357674 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357677 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357680 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.366493 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357682 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357685 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357689 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357694 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357699 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357704 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357708 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357711 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357714 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357716 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357719 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357722 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357724 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357727 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357729 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357732 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357735 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357737 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357740 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.367086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357742 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357745 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357747 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357750 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357752 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357755 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357758 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357761 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357763 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357766 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357770 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357775 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357779 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357784 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357789 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357792 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357794 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357797 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357800 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357802 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.367790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357805 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357807 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357810 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357813 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357816 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357820 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357823 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357825 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357828 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357831 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357833 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357836 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357838 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357841 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357843 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357845 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357849 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357854 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357859 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.368582 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357865 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357870 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357873 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357875 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357878 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357881 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357883 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357886 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357888 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357891 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357894 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357896 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357899 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357902 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357904 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357907 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357910 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357913 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357916 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357919 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.369216 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357921 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357924 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357926 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357929 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357934 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357939 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357946 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357951 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357954 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357956 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357960 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357964 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357967 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357970 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.357973 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.369726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.358001 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358037 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358041 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358044 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358047 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358050 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358053 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358055 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358058 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358061 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358063 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358066 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358069 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358071 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358074 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358077 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358079 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358082 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358084 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.370090 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358087 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358090 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358092 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358095 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358097 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358101 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358104 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358107 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358109 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358112 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358114 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358117 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358120 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358123 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358126 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358129 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358132 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358135 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358137 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358140 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.370683 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358143 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358145 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358148 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358151 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358153 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358157 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358160 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358163 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358166 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358168 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358171 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358173 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358176 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358179 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358181 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358184 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358186 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358191 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358194 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.371337 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358197 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358199 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358202 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358206 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358209 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358212 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358215 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358218 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358221 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358224 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358226 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358229 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358232 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358234 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358237 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358240 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358242 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358245 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358248 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358250 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.371855 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358253 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358256 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358258 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358261 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358263 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358266 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358268 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358271 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358274 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358276 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358280 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358284 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358288 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358292 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358294 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358297 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358300 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358302 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358305 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.372329 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358310 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358313 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358316 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358318 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358321 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358324 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358327 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358330 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358332 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.358335 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.361861 5036 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.361878 5036 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.361895 5036 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.361900 5036 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361965 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361971 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361975 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361979 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361982 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.372812 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361985 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361988 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361991 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361994 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.361997 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362000 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362003 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362005 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362008 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362011 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362015 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362019 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362022 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362025 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362027 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362030 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362033 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362036 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362039 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.373291 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362042 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362045 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362048 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362051 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362054 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362057 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362060 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362062 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362066 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362071 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362074 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362076 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362079 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362082 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362084 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362087 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362089 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362092 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362095 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.373774 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362098 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362100 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362103 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362105 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362108 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362111 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362113 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362116 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362118 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362121 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362123 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362126 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362129 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362131 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362134 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362137 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362139 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362142 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362144 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362147 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.374246 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362150 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362152 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362155 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362158 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362160 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362163 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362165 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362168 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362171 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362173 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362176 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362179 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362182 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362184 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362187 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362189 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362192 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362194 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362197 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.374740 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362199 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362202 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362204 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362207 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362209 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362212 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362214 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362217 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362219 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362222 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362224 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362227 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362229 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362232 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362235 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362238 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362240 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362243 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362245 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362248 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.375218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362250 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362252 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362255 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362258 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.362284 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362419 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362423 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362426 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362428 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362432 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362434 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362437 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362440 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362459 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362462 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362464 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.375939 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362467 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362470 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362472 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362475 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362478 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362480 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362482 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362485 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362487 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362490 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362494 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362497 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362499 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362502 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362505 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362507 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362510 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362513 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362515 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362518 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.376328 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362521 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362524 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362526 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362529 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362531 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362534 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362537 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362539 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362542 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362544 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362547 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362549 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362552 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362555 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362557 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362560 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362562 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362565 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362567 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.376860 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362570 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362572 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362575 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362577 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362580 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362583 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362585 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362588 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362590 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362593 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362595 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362597 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362600 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362603 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362606 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362608 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362611 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362614 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362616 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362619 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.377324 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362621 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362624 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362628 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362631 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362634 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362637 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362640 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362642 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362645 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362648 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362650 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362653 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362655 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362658 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362660 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362663 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362665 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362668 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362671 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362673 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.377830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362676 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362678 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362681 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362683 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362686 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362692 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362695 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362697 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362700 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362703 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362705 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362708 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362711 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362713 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362716 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.378357 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362720 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.362743 5036 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362782 5036 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362787 5036 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362790 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362793 5036 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362796 5036 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362798 5036 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362801 5036 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362803 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362806 5036 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362808 5036 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362811 5036 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362814 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362816 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362819 5036 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:06.378742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362822 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362824 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362827 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362829 5036 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362832 5036 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362834 5036 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362837 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362841 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362844 5036 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362847 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362849 5036 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362852 5036 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362855 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362857 5036 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362860 5036 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362862 5036 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362866 5036 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362869 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362872 5036 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:06.379126 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362875 5036 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362877 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362880 5036 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362883 5036 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362885 5036 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362888 5036 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362890 5036 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362893 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362895 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362898 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362900 5036 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362903 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362905 5036 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362908 5036 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362911 5036 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362913 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362916 5036 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362919 5036 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362921 5036 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362924 5036 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:06.379646 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362928 5036 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362931 5036 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362934 5036 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362937 5036 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362939 5036 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362942 5036 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362944 5036 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362948 5036 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362951 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362956 5036 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362958 5036 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362961 5036 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362964 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362966 5036 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362969 5036 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362972 5036 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362975 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362977 5036 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362980 5036 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:06.380133 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362982 5036 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362985 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362987 5036 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362990 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362993 5036 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362995 5036 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.362998 5036 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363001 5036 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363003 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363006 5036 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363008 5036 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363011 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363013 5036 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363017 5036 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363020 5036 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363023 5036 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363025 5036 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363028 5036 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363031 5036 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:06.380624 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363033 5036 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363036 5036 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363040 5036 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363045 5036 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363049 5036 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363052 5036 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363055 5036 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363059 5036 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363061 5036 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363064 5036 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363067 5036 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363069 5036 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363072 5036 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363074 5036 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:06.363077 5036 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.363125 5036 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364054 5036 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364100 5036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364285 5036 server.go:1014] "Starting client certificate rotation" Nov 01 21:44:06.381647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364428 5036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364498 5036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:25 +0000 UTC" deadline="2028-06-12 07:21:21.797455384 +0000 UTC" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.364524 5036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14121h37m15.43293302s" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.365440 5036 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.366959 5036 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.368551 5036 log.go:25] "Validated CRI v1 runtime API" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.370344 5036 log.go:25] "Validated CRI v1 image API" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.371161 5036 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 21:44:06.382359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.373036 5036 fs.go:135] Filesystem UUIDs: map[116cfc5e-23a0-4d66-8912-851450b0b085:/dev/nvme0n1p4 79bda8b1-d733-4548-9b93-0579954c339e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 21:44:06.382659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.373055 5036 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49d4013610c4835099eb881e206ba41b36b02b91d216764e9c59d6738c7ffd57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49d4013610c4835099eb881e206ba41b36b02b91d216764e9c59d6738c7ffd57/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f2c9646582cdb9fe72f01372df1dc23c46b5fd1e78c99ce758ae622ad299510/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f2c9646582cdb9fe72f01372df1dc23c46b5fd1e78c99ce758ae622ad299510/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fab3e0e022816a4226308666f7df3d80173a21c9b52599044c53c0d70be9044/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fab3e0e022816a4226308666f7df3d80173a21c9b52599044c53c0d70be9044/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8da50ec4f893be99d3fb74d9614ca4d46282180539786f497bbc54eb64ae85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8da50ec4f893be99d3fb74d9614ca4d46282180539786f497bbc54eb64ae85/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~projected/kube-api-access-9rsn4:{mountpoint:/var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~projected/kube-api-access-9rsn4 major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b5cff7d-1650-41de-9984-fd5cbd4b3fea/volumes/kubernetes.io~projected/kube-api-access-nhfpw:{mountpoint:/var/lib/kubelet/pods/0b5cff7d-1650-41de-9984-fd5cbd4b3fea/volumes/kubernetes.io~projected/kube-api-access-nhfpw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~projected/kube-api-access-xrpx4:{mountpoint:/var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~projected/kube-api-access-xrpx4 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab/volumes/kubernetes.io~projected/kube-api-access-wrnjg:{mountpoint:/var/lib/kubelet/pods/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab/volumes/kubernetes.io~projected/kube-api-access-wrnjg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/342e82fa-1f59-4454-beb0-d9807dc3d674/volumes/kubernetes.io~projected/kube-api-access-lth6h:{mountpoint:/var/lib/kubelet/pods/342e82fa-1f59-4454-beb0-d9807dc3d674/volumes/kubernetes.io~projected/kube-api-access-lth6h major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8/volumes/kubernetes.io~projected/kube-api-access-sfw67:{mountpoint:/var/lib/kubelet/pods/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8/volumes/kubernetes.io~projected/kube-api-access-sfw67 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c671f8-9973-4f01-a790-c286bb2ee732/volumes/kubernetes.io~projected/kube-api-access-mxr9h:{mountpoint:/var/lib/kubelet/pods/36c671f8-9973-4f01-a790-c286bb2ee732/volumes/kubernetes.io~projected/kube-api-access-mxr9h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~projected/kube-api-access-6rwts:{mountpoint:/var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~projected/kube-api-access-6rwts major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~projected/kube-api-access-s9nbl:{mountpoint:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~projected/kube-api-access-s9nbl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb4e811-66fe-409f-8a75-de7e9e9f936f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3cb4e811-66fe-409f-8a75-de7e9e9f936f/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb0e1e5-b593-4c2b-9950-ae9c967710e1/volumes/kubernetes.io~projected/kube-api-access-w4c9c:{mountpoint:/var/lib/kubelet/pods/3eb0e1e5-b593-4c2b-9950-ae9c967710e1/volumes/kubernetes.io~projected/kube-api-access-w4c9c major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/403cfda3-7bce-4dfa-bb72-b7f9f26568d0/volumes/kubernetes.io~projected/kube-api-access-gc7v9:{mountpoint:/var/lib/kubelet/pods/403cfda3-7bce-4dfa-bb72-b7f9f26568d0/volumes/kubernetes.io~projected/kube-api-access-gc7v9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/490c53c8-ef5b-4296-80b5-32a3204b483f/volumes/kubernetes.io~projected/kube-api-access-7wxrd:{mountpoint:/var/lib/kubelet/pods/490c53c8-ef5b-4296-80b5-32a3204b483f/volumes/kubernetes.io~projected/kube-api-access-7wxrd major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~projected/kube-api-access-rdpb5:{mountpoint:/var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~projected/kube-api-access-rdpb5 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6777c192-94cb-4df3-8ed7-3ce7688c0ba7/volumes/kubernetes.io~projected/kube-api-access-bn7vn:{mountpoint:/var/lib/kubelet/pods/6777c192-94cb-4df3-8ed7-3ce7688c0ba7/volumes/kubernetes.io~projected/kube-api-access-bn7vn major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/kube-api-access-kbsxl:{mountpoint:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/kube-api-access-kbsxl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793bff98-0107-4731-a6a0-5747f9303443/volumes/kubernetes.io~projected/kube-api-access-2l4dh:{mountpoint:/var/lib/kubelet/pods/793bff98-0107-4731-a6a0-5747f9303443/volumes/kubernetes.io~projected/kube-api-access-2l4dh major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a4912cf-de1f-4070-9635-47de5fa95ad5/volumes/kubernetes.io~projected/kube-api-access-hmccv:{mountpoint:/var/lib/kubelet/pods/7a4912cf-de1f-4070-9635-47de5fa95ad5/volumes/kubernetes.io~projected/kube-api-access-hmccv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~projected/kube-api-access-mkjhd:{mountpoint:/var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~projected/kube-api-access-mkjhd major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89d6f94c-7d2b-420b-bea6-0bb929dd6194/volumes/kubernetes.io~projected/kube-api-access-j8rjb:{mountpoint:/var/lib/kubelet/pods/89d6f94c-7d2b-420b-bea6-0bb929dd6194/volumes/kubernetes.io~projected/kube-api-access-j8rjb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6c0a65-773b-45c9-b18b-bd582d7a6307/volumes/kubernetes.io~projected/kube-api-access-9skmp:{mountpoint:/var/lib/kubelet/pods/8e6c0a65-773b-45c9-b18b-bd582d7a6307/volumes/kubernetes.io~projected/kube-api-access-9skmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91699fd6-b0b6-45c7-8d3c-5175c3a30979/volumes/kubernetes.io~projected/kube-api-access-ltgk8:{mountpoint:/var/lib/kubelet/pods/91699fd6-b0b6-45c7-8d3c-5175c3a30979/volumes/kubernetes.io~projected/kube-api-access-ltgk8 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a47e4a36-2064-44c4-bf33-1c26e5b7603c/volumes/kubernetes.io~projected/kube-api-access-r49cn:{mountpoint:/var/lib/kubelet/pods/a47e4a36-2064-44c4-bf33-1c26e5b7603c/volumes/kubernetes.io~projected/kube-api-access-r49cn major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~projected/kube-api-access-ml2nq:{mountpoint:/var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~projected/kube-api-access-ml2nq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/kube-api-access-9mp6v:{mountpoint:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/kube-api-access-9mp6v major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~projected/kube-api-access-csf7t:{mountpoint:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~projected/kube-api-access-csf7t major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bef85c8b-3d71-48ea-95f5-9104415811b2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bef85c8b-3d71-48ea-95f5-9104415811b2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd265d9d-2558-4f1c-8b67-545bf442aa7c/volumes/kubernetes.io~projected/kube-api-access-kfq5p:{mountpoint:/var/lib/kubelet/pods/cd265d9d-2558-4f1c-8b67-545bf442aa7c/volumes/kubernetes.io~projected/kube-api-access-kfq5p major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cef8eaf9-2d56-4d3f-a171-3583becf435e/volumes/kubernetes.io~projected/kube-api-access-zbmxc:{mountpoint:/var/lib/kubelet/pods/cef8eaf9-2d56-4d3f-a171-3583becf435e/volumes/kubernetes.io~projected/kube-api-access-zbmxc major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfaa13d7-7690-4203-842a-94b26ba0331b/volumes/kubernetes.io~projected/kube-api-access-8bdbf:{mountpoint:/var/lib/kubelet/pods/cfaa13d7-7690-4203-842a-94b26ba0331b/volumes/kubernetes.io~projected/kube-api-access-8bdbf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9ae9063-f7f1-450b-adbf-b21c148dea72/volumes/kubernetes.io~projected/kube-api-access-6bz8g:{mountpoint:/var/lib/kubelet/pods/d9ae9063-f7f1-450b-adbf-b21c148dea72/volumes/kubernetes.io~projected/kube-api-access-6bz8g major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~projected/kube-api-access-lwxjq:{mountpoint:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~projected/kube-api-access-lwxjq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fddb723d-930f-4b42-8f63-9780761c0119/volumes/kubernetes.io~projected/kube-api-access-gzcqf:{mountpoint:/var/lib/kubelet/pods/fddb723d-930f-4b42-8f63-9780761c0119/volumes/kubernetes.io~projected/kube-api-access-gzcqf major:0 minor:226 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a11f489b2b3573ec671841b9afb4c79df8fd01fab12675bd8474b1c1af0bba40/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d368ca730777cdf9044584beafcab4e0abea2c8a60447235fcfdc552d9235642/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0990df7f3158ff369d247c7356d8aa2bdccdc7688854f42622ddc621c42fe60a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/d74114ca307e3892796ebb9661c0cdebb09e227e8f27654c48260353dbd0ae66/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7434ae0b96aae705b1b27e8c5da05b2e7e1e9783efc5a339a00dcb408e80b73f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/58f9b853f09b70c00d21ce15aa661f873d47eefba3e5f7440170cfe590a4f006/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5d3269ebf77b37401618f57bba45cafde6a59d4fe9163081a37de4acc14a8b41/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/177fa2ee67658ed3779477fe7650bb53ffbe9c598927e79074105203399f87f4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ca7a12f8dd111f2f869ac5fbfe2ccbec3e48cd3d9d58ea556ca0853bc17dfe90/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/de2445684216f952fd5bd200d77bf510e2666937e5d5995a7ed3cc3396cc1959/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/acb7e03ca3503b6cead581f1968d60408463cac8ca158080e99d5ff0d3bd6552/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5381ce30172a68c8f4921095713080a2f803d0313a60f69a0d0c19990fccae1e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c746664e6641d454514ad824c39f13324fb5262ccd336a12928936c5efee2a0e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5b566cd4e76a0f05616161d63954ea752fc518b83673166b632f1dde6e261450/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/23b5ebeafa9d77a20afca94944b25db1c635755dea14675b19a983568d8da6f9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/febee680d33229db4fe9c67402471101b62cf624243fd965991d79a8b14c1d94/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/a5e447081f5ceecfdf8f0958b4592f2a162e2ab65987e347997b66c7222c37d1/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5723fcc2b119f3d4491623b5e9c7726849a960a8c24178e0a71be4caba531d02/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e5abdcc05d4e17d172ff8c49b28096d61f8807585efe4109230b309784140f43/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/01e27baef51c1bcc5c08b64543f97752b3ec0a39505f1405491a3127f7f0bbe4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/80d371607e4f5019006a01a89d44c284c1aaef04c2da4feac82e1524f6e2ca4a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/84b98d94f5d6440ec0c7025e86cc15da7b379d8dc79030c2b98affd827636a9e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ee0b22fd9641aa6f6b4f9636329e672d9421c7cd9327578e82a62256f92a47fc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/81a414f5c3373cb331b42ea748a5bdc91955e485cc4622a750c05d8455eaf780/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/cb05ed9c479b424facd8a63e032be5248dd268921bd404b1f9283aeaeaa10a6b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fad38f9cf437170857d20943295714278c7652fe1b45c078a77d694583616a43/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/bdbeabedec6f607cdcfb8d5902f768566e3f43db852b06dee47c8969a4481e0d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/dcd36689dedbdb5f7fc973ae8381a9e6b74df8f059a9c1f344c4b5b018be4293/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b3f1971d9e82eae699d3030dde08e7ce99038beca3049fee720cb67776df76c4/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/983a0c41d05daa17897067327e7c3c729c5fba0a56914176951ae36a97f3c037/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/00fe8f0cca969b22d9dcee04e23080f595e8c023a1a749dabf59fb46d5a130d6/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/37d90a0e01ac457e16941e7f4db21b7418cec06a3e51a1eaf1a7253ca3c31797/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/c19e5b8faa7a9a6fba6b54c6ee2ac6620d8765ca7836d6c3664a81813c94d534/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/67d1097d0043d42f13d0d284bd5f9e4e7e157477447981f74004c3b7f7e2b0c1/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/cc52eb730b9c3be8adee0d4302187a435cda97a5ec610b2a8f927ed8d8b00804/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d35e17a21407e67100a9e208f3965d6f9c55e05510770b1002cb3ec707717b3a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/38bf5968ed6fa9d456d91df86987182ce57c3f88a8c83e4d77ebd4e7feeddf62/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/348d4ed3bcadff11ca801b89eb39321817a6cd7b45c7169cecf6a5093c39cad7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/28488eb077a802a5d90dcffc3d28fa42420166fe264ee1d9a873ef809c79440b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/912884c3c6314c648ee72adc87017fedbafbe721e815623f1ddc3fba9a2fa5fb/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/43514515c853310137362460a5c4c9a263dff92b4215a17e63dd051941e7b4ce/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/1e7039aa2c4808c022ee116ad0fb537cd3cb3f53ef976df9aba2662fffa64269/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/63ea72f31026922780a5fc1142f722edee3c5395b0f78e7ce3b90b7a8325ac8d/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/b139fbf94bda69f7bc9d234384ef5531aade76750161e865f4d56d8721bba2e8/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/613456687f1fb1761752985ad8af67aa08b620412298149eeffaf4568e673ff3/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/9e06e376af1ea067ea42da0c81d2790ae7cbcae8d8d08eb385b137bb1167ee2f/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/2dd34754c605c9454cd06e1957972663c5ba28adf4d622982efdea328dbf9f40/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/36a3810f95d72f8fcba131952360b175fa10162b7a63a64b1ef4c36175699f0f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/afdf000137f78c6b3405162c565b346a96ff65ea2115baa70f5adfd86f674a03/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/8860497f84dac4852fc6a21f865534133a33028c5035fa6e7ea0f7f8e96bf93f/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/de11947f08ceb9afa5b949a2ecdd543c0a558583447bd845848c4d8b97f1a77f/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/1ddad12d47c1b92482273592340c66376b7152ffb311aa5b785f57836a45e16a/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a3b977c60092432f9ce1bbece87dcf0f662a8a806c2b5374a501cd493dd95f66/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/479a748bf13711641f884307ed5cbc4d363ad8c1c9293fee9fe5681a69f43dbf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d9b85219e6f9a69d69e4c8974a5bf6bc12572b5e715275defe1cc11baa9710b1/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8c7a28000e1047041764cc9cb06a2e75e579cfb24f8910bd69bee565d01e31b5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/04004f153c3a7cfc78a28bbfd576d78334d6178960d146dceec827ba978bdd94/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/99554a52a40fa804b0ee76e3f0f5ef10b991bb4d2dec45846650916bfa8746ab/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7371d3ed2838ee8e738f0679127e212c06d0c746ac5de241fce700b25432f000/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6af9f055f4bc0a58ce4623711c057c2737554370249d7d26aff9b3e255e68f0e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8cc36234303c467196a282e67b73b7480ef6784ecc12645ba985e08d8daf3c00/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/8baa0a71c12a8f881e2efdb758bf7abb5b2b90d148e66f1601e11ab383df0ac0/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.380850 5036 manager.go:217] Machine: {Timestamp:2026-11-01 21:44:06.38024383 +0000 UTC m=+0.073224307 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092387 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec257749b6a70b71afd90a35ad161251 SystemUUID:ec257749-b6a7-0b71-afd9-0a35ad161251 BootID:f493d2a3-6977-4a8b-ab5b-39484d6adca3 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3882b4c7c4b0b0f0e7dc9388fbfbb4e6aeaf7ae85703d79f5f110a94c1cc0806/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~projected/kube-api-access-mkjhd DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bef85c8b-3d71-48ea-95f5-9104415811b2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/kube-api-access-9mp6v DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~projected/kube-api-access-6rwts DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ff1b01fe25e98850b2a5979be62ae97025d632e4dd803e022e75b139d60f63/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6c0a65-773b-45c9-b18b-bd582d7a6307/volumes/kubernetes.io~projected/kube-api-access-9skmp DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~projected/kube-api-access-rdpb5 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fddb723d-930f-4b42-8f63-9780761c0119/volumes/kubernetes.io~projected/kube-api-access-gzcqf DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f2c9646582cdb9fe72f01372df1dc23c46b5fd1e78c99ce758ae622ad299510/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab/volumes/kubernetes.io~projected/kube-api-access-wrnjg DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36c671f8-9973-4f01-a790-c286bb2ee732/volumes/kubernetes.io~projected/kube-api-access-mxr9h DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~projected/kube-api-access-ml2nq DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08b93ccd182e56d9ea8acb6df0901bf0ebc7cc65b57eca6d073aeb189fd74cd7/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62484713c54f264c1d1beb7c54b4bd48fe69c24576f819e715052f79b914a035/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b039cbf4098bcdd774dd42328d62236c2843e259bba2fecdd6a91f9cdb149fc3/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6eac2e089f452a4e782fe04e945a09789fc9515e8b35b5d63912d9bbc55cc43/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b5cff7d-1650-41de-9984-fd5cbd4b3fea/volumes/kubernetes.io~projected/kube-api-access-nhfpw DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~projected/kube-api-access-kbsxl DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7acbcc79581c38d320b159ad9bc75b48f7c391eead6da5bcf628e0dae3cd2d5e/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a47e4a36-2064-44c4-bf33-1c26e5b7603c/volumes/kubernetes.io~projected/kube-api-access-r49cn DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8da50ec4f893be99d3fb74d9614ca4d46282180539786f497bbc54eb64ae85/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d7b6c1cd874b3eb02e978004c7e8f3d32cdfc2b2d41c245d99d39503d84c905/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f669fe8fed34573f9868cb59b4db8cea57fe39076887c2d806ce85cd8b02115d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/833deb27-8ec7-4e48-9827-a627d0c7fd5c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6777c192-94cb-4df3-8ed7-3ce7688c0ba7/volumes/kubernetes.io~projected/kube-api-access-bn7vn DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb4e811-66fe-409f-8a75-de7e9e9f936f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/793bff98-0107-4731-a6a0-5747f9303443/volumes/kubernetes.io~projected/kube-api-access-2l4dh DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/342e82fa-1f59-4454-beb0-d9807dc3d674/volumes/kubernetes.io~projected/kube-api-access-lth6h DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e632f276-6eef-4478-8c2c-1f7e4b98a6d7/volumes/kubernetes.io~projected/kube-api-access-lwxjq DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/510010af-f8ad-4b48-9f73-e9ad1b0a2de9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49d4013610c4835099eb881e206ba41b36b02b91d216764e9c59d6738c7ffd57/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29ac9eed0a4fd9c610a87eb79fdb79c256b124331ea0056821bc2dfe63bf66aa/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0693fb2b-eba1-4b24-a44b-1a98ddfedd54/volumes/kubernetes.io~projected/kube-api-access-9rsn4 DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91699fd6-b0b6-45c7-8d3c-5175c3a30979/volumes/kubernetes.io~projected/kube-api-access-ltgk8 DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/966e7383d2f013937a07993ab530210bfd8b7216168d545ce01c7fd22c26b1c3/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2e2059c1016ec967bec7844a73ab9f2613c549136c1e016199fb96f81462ce/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c84a730bdd65dde4aa7acc46069bde3ec7575c1556a0b6add17147606bc48e7a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d640e8f941a05a53a08dbfe4eb6a134b5c753ffcaec16c5c0101c7110771ecda/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfaa13d7-7690-4203-842a-94b26ba0331b/volumes/kubernetes.io~projected/kube-api-access-8bdbf DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/403cfda3-7bce-4dfa-bb72-b7f9f26568d0/volumes/kubernetes.io~projected/kube-api-access-gc7v9 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a79cdc8f-6b2d-4623-b321-be0088cfad25/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f3235b8cb01e502b3dc443ba14c27f22f3ce00abc3e7759d843b1f593044fe0/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7a4912cf-de1f-4070-9635-47de5fa95ad5/volumes/kubernetes.io~projected/kube-api-access-hmccv DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d43339b545ba3c537b9d2fd298c55d9f852058f2b336f8eda448fe2a2fee0e2/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/490c53c8-ef5b-4296-80b5-32a3204b483f/volumes/kubernetes.io~projected/kube-api-access-7wxrd DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fab3e0e022816a4226308666f7df3d80173a21c9b52599044c53c0d70be9044/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/329bf65cf264936bdb02df6b521c4b38bf911cef224dc46c771416aec78710a6/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cef8eaf9-2d56-4d3f-a171-3583becf435e/volumes/kubernetes.io~projected/kube-api-access-zbmxc DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd265d9d-2558-4f1c-8b67-545bf442aa7c/volumes/kubernetes.io~projected/kube-api-access-kfq5p DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/385fff43-5171-48dd-8530-98bdfb742bec/volumes/kubernetes.io~projected/kube-api-access-s9nbl DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f3b2036-627e-4704-91fb-87df2b35c328/volumes/kubernetes.io~projected/kube-api-access-xrpx4 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/305fdb61af9b90c1d1da7d3804d70a75c9e1769db388e490e318a2276839eae8/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~projected/kube-api-access-csf7t DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae24d6fe19c3960b44506b318af130e206fefb5499127b01576c0e98bebec76c/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9ae9063-f7f1-450b-adbf-b21c148dea72/volumes/kubernetes.io~projected/kube-api-access-6bz8g DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb0e1e5-b593-4c2b-9950-ae9c967710e1/volumes/kubernetes.io~projected/kube-api-access-w4c9c DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89d6f94c-7d2b-420b-bea6-0bb929dd6194/volumes/kubernetes.io~projected/kube-api-access-j8rjb DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd85b460-2a23-473f-98c5-757f7dc2b421/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8/volumes/kubernetes.io~projected/kube-api-access-sfw67 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08b93ccd182e56d MacAddress:5e:84:4c:5b:59:6f Speed:10000 Mtu:8901} {Name:29ac9eed0a4fd9c MacAddress:6e:51:9c:bc:d2:e5 Speed:10000 Mtu:8901} {Name:305fdb61af9b90c MacAddress:02:17:59:a4:c4:0a Speed:10000 Mtu:8901} {Name:307de04b58832f2 MacAddress:12:8f:78:8f:2e:38 Speed:10000 Mtu:8901} {Name:49d4013610c4835 MacAddress:fe:6a:67:81:e0:21 Speed:10000 Mtu:8901} {Name:4e2e2059c1016ec MacAddress:4a:d7:fd:c6:cc:ab Speed:10000 Mtu:8901} {Name:5d43339b545ba3c MacAddress:62:cd:4c:58:32:d1 Speed:10000 Mtu:8901} {Name:62484713c54f264 MacAddress:c2:52:a9:84:f3:cb Speed:10000 Mtu:8901} {Name:7acbcc79581c38d MacAddress:aa:65:8e:37:b2:8d Speed:10000 Mtu:8901} {Name:80ff1b01fe25e98 MacAddress:7e:b1:99:50:cc:34 Speed:10000 Mtu:8901} {Name:8f3235b8cb01e50 MacAddress:32:78:6e:93:7c:0a Speed:10000 Mtu:8901} {Name:966e7383d2f0139 MacAddress:a2:15:56:db:6d:37 Speed:10000 Mtu:8901} {Name:ae24d6fe19c3960 MacAddress:66:00:9b:20:57:c7 Speed:10000 Mtu:8901} {Name:b039cbf4098bcdd MacAddress:96:ac:59:1e:38:91 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ce:f6:24:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:56:c0:7e:b9:4d:0d Speed:0 Mtu:8901} {Name:d640e8f941a05a5 MacAddress:6e:8a:25:70:57:2c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ce:f6:24:51 Speed:0 Mtu:9001} {Name:f6eac2e089f452a MacAddress:8a:34:d6:ff:0a:78 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:13:91:d0:9f:a1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:01:b7:cf:99:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.381807 5036 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.381899 5036 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382417 5036 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382437 5036 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-110.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382574 5036 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382580 5036 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382593 5036 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382610 5036 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382630 5036 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382783 5036 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382880 5036 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382910 5036 kubelet.go:493] "Attempting to sync node with API server" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382919 5036 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382932 5036 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382941 5036 kubelet.go:405] "Adding apiserver pod source" Nov 01 21:44:06.382934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.382949 5036 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 21:44:06.383958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.383727 5036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 21:44:06.383958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.383739 5036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:44:06.384225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.384201 5036 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 21:44:06.387185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387169 5036 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 21:44:06.387185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387183 5036 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:44:06.387299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387289 5036 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 21:44:06.387533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387520 5036 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 21:44:06.387574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387551 5036 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 21:44:06.387655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387646 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387660 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387666 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387673 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387681 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387690 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 21:44:06.387695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387697 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387703 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387709 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387715 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387729 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387738 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387762 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 21:44:06.387881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.387771 5036 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 21:44:06.391273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.391258 5036 server.go:1263] "Started kubelet" Nov 01 21:44:06.391899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.391395 5036 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 21:44:06.391899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.391500 5036 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 21:44:06.391792 ip-10-0-131-110 systemd[1]: Started Kubernetes Kubelet. Nov 01 21:44:06.392101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.391924 5036 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 21:44:06.392891 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.391883 5036 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 21:44:06.393929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.393909 5036 server.go:317] "Adding debug handlers to kubelet server" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398560 5036 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398847 5036 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398861 5036 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.398868 5036 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-110.us-east-2.compute.internal\" not found" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398874 5036 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398975 5036 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.398968 5036 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:26 +0000 UTC" deadline="2028-05-08 11:35:51.186995977 +0000 UTC" Nov 01 21:44:06.399187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.399023 5036 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13285h51m44.787977185s" Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402552 5036 factory.go:153] Registering CRI-O factory Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402573 5036 factory.go:223] Registration of the crio container factory successfully Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402624 5036 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402633 5036 factory.go:55] Registering systemd factory Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402644 5036 factory.go:223] Registration of the systemd container factory successfully Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402660 5036 factory.go:103] Registering Raw factory Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.402674 5036 manager.go:1199] Started watching for new ooms in manager Nov 01 21:44:06.403475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.403099 5036 manager.go:319] Starting recovery of all containers Nov 01 21:44:06.405340 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405288 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a4912cf-de1f-4070-9635-47de5fa95ad5" volumeName="kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405347 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd85b460-2a23-473f-98c5-757f7dc2b421" volumeName="kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405362 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9ae9063-f7f1-450b-adbf-b21c148dea72" volumeName="kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405375 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793bff98-0107-4731-a6a0-5747f9303443" volumeName="kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405399 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79cdc8f-6b2d-4623-b321-be0088cfad25" volumeName="kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405411 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfaa13d7-7690-4203-842a-94b26ba0331b" volumeName="kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf" seLinuxMountContext="" Nov 01 21:44:06.405431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405424 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" volumeName="kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405459 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91699fd6-b0b6-45c7-8d3c-5175c3a30979" volumeName="kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405473 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a47e4a36-2064-44c4-bf33-1c26e5b7603c" volumeName="kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405493 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd85b460-2a23-473f-98c5-757f7dc2b421" volumeName="kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405517 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833deb27-8ec7-4e48-9827-a627d0c7fd5c" volumeName="kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405529 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" volumeName="kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405541 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405573 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89d6f94c-7d2b-420b-bea6-0bb929dd6194" volumeName="kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405600 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6c0a65-773b-45c9-b18b-bd582d7a6307" volumeName="kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405612 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405648 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e632f276-6eef-4478-8c2c-1f7e4b98a6d7" volumeName="kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405662 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405709 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342e82fa-1f59-4454-beb0-d9807dc3d674" volumeName="kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405732 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c671f8-9973-4f01-a790-c286bb2ee732" volumeName="kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script" seLinuxMountContext="" Nov 01 21:44:06.405772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405747 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6777c192-94cb-4df3-8ed7-3ce7688c0ba7" volumeName="kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405786 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793bff98-0107-4731-a6a0-5747f9303443" volumeName="kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405817 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79cdc8f-6b2d-4623-b321-be0088cfad25" volumeName="kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405854 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405867 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342e82fa-1f59-4454-beb0-d9807dc3d674" volumeName="kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405887 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" volumeName="kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405899 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405910 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793bff98-0107-4731-a6a0-5747f9303443" volumeName="kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405935 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405947 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b5cff7d-1650-41de-9984-fd5cbd4b3fea" volumeName="kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405966 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405985 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.405996 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cef8eaf9-2d56-4d3f-a171-3583becf435e" volumeName="kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406007 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e632f276-6eef-4478-8c2c-1f7e4b98a6d7" volumeName="kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406025 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c671f8-9973-4f01-a790-c286bb2ee732" volumeName="kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406037 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37cb1d59-4690-49c0-95d5-ecf9ce11ff7d" volumeName="kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406049 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb4e811-66fe-409f-8a75-de7e9e9f936f" volumeName="kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406068 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="403cfda3-7bce-4dfa-bb72-b7f9f26568d0" volumeName="kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406102 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" volumeName="kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406115 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89d6f94c-7d2b-420b-bea6-0bb929dd6194" volumeName="kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406127 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6c0a65-773b-45c9-b18b-bd582d7a6307" volumeName="kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406138 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693fb2b-eba1-4b24-a44b-1a98ddfedd54" volumeName="kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406149 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385fff43-5171-48dd-8530-98bdfb742bec" volumeName="kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406160 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" volumeName="kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406170 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a79cdc8f-6b2d-4623-b321-be0088cfad25" volumeName="kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406184 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406198 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bef85c8b-3d71-48ea-95f5-9104415811b2" volumeName="kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406209 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddb723d-930f-4b42-8f63-9780761c0119" volumeName="kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406220 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693fb2b-eba1-4b24-a44b-1a98ddfedd54" volumeName="kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406231 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30fdcebb-da92-461b-a1a1-1ab5e46ec0ab" volumeName="kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406241 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833deb27-8ec7-4e48-9827-a627d0c7fd5c" volumeName="kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406253 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406291 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e632f276-6eef-4478-8c2c-1f7e4b98a6d7" volumeName="kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406303 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385fff43-5171-48dd-8530-98bdfb742bec" volumeName="kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406316 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385fff43-5171-48dd-8530-98bdfb742bec" volumeName="kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406337 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="490c53c8-ef5b-4296-80b5-32a3204b483f" volumeName="kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406356 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" volumeName="kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406367 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd85b460-2a23-473f-98c5-757f7dc2b421" volumeName="kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406380 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb4e811-66fe-409f-8a75-de7e9e9f936f" volumeName="kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406391 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6777c192-94cb-4df3-8ed7-3ce7688c0ba7" volumeName="kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406402 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406413 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406429 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406456 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a4912cf-de1f-4070-9635-47de5fa95ad5" volumeName="kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406469 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385fff43-5171-48dd-8530-98bdfb742bec" volumeName="kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406479 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693fb2b-eba1-4b24-a44b-1a98ddfedd54" volumeName="kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406491 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" volumeName="kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406502 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="403cfda3-7bce-4dfa-bb72-b7f9f26568d0" volumeName="kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406513 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" volumeName="kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca" seLinuxMountContext="" Nov 01 21:44:06.406490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406524 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406535 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406546 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="342e82fa-1f59-4454-beb0-d9807dc3d674" volumeName="kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406556 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37cb1d59-4690-49c0-95d5-ecf9ce11ff7d" volumeName="kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406567 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385fff43-5171-48dd-8530-98bdfb742bec" volumeName="kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406577 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="403cfda3-7bce-4dfa-bb72-b7f9f26568d0" volumeName="kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406587 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="521c4405-c6bb-4fd3-bcfb-3cad016376e2" volumeName="kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406599 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833deb27-8ec7-4e48-9827-a627d0c7fd5c" volumeName="kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406609 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406621 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30fdcebb-da92-461b-a1a1-1ab5e46ec0ab" volumeName="kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406632 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3665c82e-f3a8-461e-bb7a-d4cb6d6111a8" volumeName="kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406643 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406654 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3b56e-3966-4258-8169-b20577969634" volumeName="kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406665 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd265d9d-2558-4f1c-8b67-545bf442aa7c" volumeName="kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406676 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f3b2036-627e-4704-91fb-87df2b35c328" volumeName="kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406688 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ec954c-cf47-42ac-86f8-0daab114e5f2" volumeName="kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406698 5036 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793bff98-0107-4731-a6a0-5747f9303443" volumeName="kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh" seLinuxMountContext="" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406708 5036 reconstruct.go:97] "Volume reconstruction finished" Nov 01 21:44:06.409066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.406717 5036 reconciler.go:29] "Reconciler: start to sync state" Nov 01 21:44:06.413928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.413910 5036 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:06.414205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.414179 5036 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:06.414890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.414861 5036 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 21:44:06.414979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.414963 5036 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:06.421190 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.421172 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:06.421518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.421492 5036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 21:44:06.422326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.422306 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:06.429828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.429808 5036 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 21:44:06.429828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.429829 5036 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 21:44:06.429978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.429845 5036 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 21:44:06.429978 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.429881 5036 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 21:44:06.434153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.434135 5036 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:06.450379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.450356 5036 generic.go:356] "Generic (PLEG): container finished" podID="571993d965c8e1c01034d44492f7be15" containerID="f1e58b37b7c135d9cdce6b409e21deabb962d93d346b54e14ed36c7fa228cc71" exitCode=0 Nov 01 21:44:06.460344 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.460328 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:06.460426 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.460355 5036 generic.go:356] "Generic (PLEG): container finished" podID="510010af-f8ad-4b48-9f73-e9ad1b0a2de9" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" exitCode=255 Nov 01 21:44:06.473674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473643 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="1ef7fb136498fac797e549fe561c8c95529c4c1b8f7d4a11af718bd396e263dd" exitCode=0 Nov 01 21:44:06.473674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473664 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="e4885f2c3a4b1235b8c6f6ec9fa007b087ef88ce89dec89c62815db69fafca05" exitCode=0 Nov 01 21:44:06.473674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473674 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="deaea5266bb46a164d4ec3ec180ff50f079398b478f5b2ea66d38fe2cf89072f" exitCode=0 Nov 01 21:44:06.473839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473680 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="b4735035d86b01d7ac3a6e4171e150e7cc252afc3425445f39c63dbb01508ddf" exitCode=0 Nov 01 21:44:06.473839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473685 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="f7c51de886ef1de1dfeb56c6f4e9eac770f56dcebd8245afd84e6adfd670b35c" exitCode=0 Nov 01 21:44:06.473839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.473693 5036 generic.go:356] "Generic (PLEG): container finished" podID="3eb0e1e5-b593-4c2b-9950-ae9c967710e1" containerID="246a9a09b8ad9def32e3c45e57ac4df01027d357893d3d4cd7d1d99b23a40566" exitCode=0 Nov 01 21:44:06.502018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.502000 5036 manager.go:324] Recovery completed Nov 01 21:44:06.523348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523325 5036 cpu_manager.go:226] "Starting" policy="none" Nov 01 21:44:06.523348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523342 5036 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 21:44:06.523470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523360 5036 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 21:44:06.523512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523498 5036 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523513 5036 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523527 5036 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523533 5036 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523538 5036 policy_none.go:50] "Start" Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523546 5036 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 21:44:06.523601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523556 5036 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 21:44:06.523766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523674 5036 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 21:44:06.523766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523681 5036 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 21:44:06.523766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.523687 5036 policy_none.go:44] "Start" Nov 01 21:44:06.528006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.527981 5036 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 21:44:06.529965 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.529949 5036 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 21:44:06.531731 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.531717 5036 manager.go:341] "Starting Device Plugin manager" Nov 01 21:44:06.532180 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.531747 5036 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 21:44:06.532180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532182 5036 server.go:87] "Starting device plugin registration server" Nov 01 21:44:06.532426 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532415 5036 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 21:44:06.532510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532426 5036 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 21:44:06.532563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532523 5036 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 21:44:06.532673 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532635 5036 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 21:44:06.532735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532685 5036 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 21:44:06.532735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532693 5036 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 21:44:06.532825 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.532748 5036 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:44:06.53267296Z","UUID":"7df8a534-3c0a-4724-bb4e-a09873cd4edb","Handler":null,"Name":"","Endpoint":""} Nov 01 21:44:06.533903 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.533879 5036 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 21:44:06.534108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.534089 5036 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 21:44:06.534196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.534125 5036 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 21:44:06.633388 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.633330 5036 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:44:06.634479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.634461 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:44:06.634570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.634492 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:44:06.634570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.634504 5036 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-110.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:44:06.634570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.634543 5036 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.647812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.647792 5036 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.647888 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.647855 5036 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.730770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.730730 5036 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal"] Nov 01 21:44:06.731336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731322 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.731374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerStarted","Data":"35b6d9fe4f4ddee886fd33e809fb87886ffeb7ce6537dd07c9ec155d7ec84eff"} Nov 01 21:44:06.731374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerDied","Data":"f1e58b37b7c135d9cdce6b409e21deabb962d93d346b54e14ed36c7fa228cc71"} Nov 01 21:44:06.731374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"571993d965c8e1c01034d44492f7be15","Type":"ContainerStarted","Data":"f8a6504afeac2c38537642253dbb85fc29edd59c3440d260191373c54726cd2e"} Nov 01 21:44:06.731488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731479 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"fc8a4eb4513497bb3852f6aab7c20b22","Type":"ContainerStarted","Data":"b717c1abf4bdbecf6d589c21838d2dd3df18db069e53dbe85af787e6a880a435"} Nov 01 21:44:06.731528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" event={"ID":"fc8a4eb4513497bb3852f6aab7c20b22","Type":"ContainerStarted","Data":"53e1d1a01dbabc5ab8fc216d7a2596bd58dac00a79f2496727a7c7b142ab66d2"} Nov 01 21:44:06.731635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.731625 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.756797 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.756779 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:06.756870 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.756826 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.756870 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.756830 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:06.756947 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:06.756867 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.817661 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.817638 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.817737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.817667 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.817737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.817687 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918160 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918160 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918139 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918160 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918202 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918210 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571993d965c8e1c01034d44492f7be15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"571993d965c8e1c01034d44492f7be15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:06.918334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:06.918218 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8a4eb4513497bb3852f6aab7c20b22-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" (UID: \"fc8a4eb4513497bb3852f6aab7c20b22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:07.383766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.383737 5036 apiserver.go:52] "Watching apiserver" Nov 01 21:44:07.396973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.396944 5036 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 21:44:07.397712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.397682 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wbmm4","openshift-insights/insights-operator-7c4986d449-lqtc9","openshift-network-console/networking-console-plugin-86c478bb88-5fkz5","openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws","openshift-insights/periodic-gathering-h4t2h-kzgn5","openshift-multus/multus-additional-cni-plugins-4tw88","openshift-cluster-node-tuning-operator/tuned-8n2mc","openshift-multus/multus-ckgds","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm","openshift-dns/dns-default-55x22","kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal","openshift-dns/node-resolver-jxl58","openshift-image-registry/image-registry-596f45d76f-hl6hx","openshift-console-operator/console-operator-7cd56b8dbc-nscn6","kube-system/kas-connection-checker-685895c7c9-6qtm8","openshift-insights/insights-runtime-extractor-4bjns","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6","openshift-network-operator/iptables-alerter-z9t47","openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn","openshift-multus/network-metrics-daemon-vlnwn","openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6","kube-system/konnectivity-agent-4tvq4","kube-system/kas-connection-checker-685895c7c9-chf9m","kube-system/kas-connection-checker-685895c7c9-lfwr9","openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf","openshift-ovn-kubernetes/ovnkube-node-lcz57","openshift-ingress/router-default-554f8bcb97-6mq5w","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7","kube-system/global-pull-secret-syncer-wr8bc","openshift-network-diagnostics/network-check-target-t7lmc","openshift-image-registry/node-ca-q2cpx","openshift-image-registry/image-registry-6c45c69888-j7vl7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj","openshift-service-ca/service-ca-5fffc97bdf-6w2xf","openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn"] Nov 01 21:44:07.398992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.398971 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.399961 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.399942 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.402008 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.401989 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.402391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.402370 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.402507 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.402424 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.402576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.402555 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 21:44:07.402994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.402979 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.407294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407248 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmv5z\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407363 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407433 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407438 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407437 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407489 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407524 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 21:44:07.407594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407581 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407655 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407696 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407727 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407798 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407838 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407937 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407433 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.407990 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408010 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.408028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408014 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408045 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408057 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jv95p\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408066 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408071 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408203 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408284 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.408540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408319 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 21:44:07.408788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.408624 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.409146 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409124 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 21:44:07.409238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409169 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.409238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409229 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 21:44:07.409339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409240 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vvt4k\"" Nov 01 21:44:07.409339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409302 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 21:44:07.409339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409303 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9glb6\"" Nov 01 21:44:07.409641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409621 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.409641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409636 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.409806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409719 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.409806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409775 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n8gq9\"" Nov 01 21:44:07.409923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.409902 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzsqs\"" Nov 01 21:44:07.410238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410220 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 21:44:07.410361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410242 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.410361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410284 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hzp42\"" Nov 01 21:44:07.410494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410321 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rcssj\"" Nov 01 21:44:07.410646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410621 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 21:44:07.410706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410659 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vwhft\"" Nov 01 21:44:07.410749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410733 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.410827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.410811 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 21:44:07.411865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.411849 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.412138 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.412119 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.412515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.412498 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 21:44:07.412580 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.412537 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 21:44:07.412710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.412688 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fpfrj\"" Nov 01 21:44:07.413808 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.413791 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 21:44:07.413976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.413959 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 21:44:07.414554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.414532 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 21:44:07.415134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.415117 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.415704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.415689 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.416183 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416163 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 21:44:07.416283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416227 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.416283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416246 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 21:44:07.416481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416457 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\"" Nov 01 21:44:07.416570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416494 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.416570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416549 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\"" Nov 01 21:44:07.416671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416611 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\"" Nov 01 21:44:07.416671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416619 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 21:44:07.416671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416548 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\"" Nov 01 21:44:07.416671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416665 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\"" Nov 01 21:44:07.416847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416737 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.416847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416468 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 21:44:07.416847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.416762 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 21:44:07.417386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417365 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.417534 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417372 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.417534 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417409 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5qkrj\"" Nov 01 21:44:07.417684 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417665 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\"" Nov 01 21:44:07.417735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417723 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\"" Nov 01 21:44:07.417852 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417835 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.417900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.417836 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 21:44:07.418530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.418510 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 21:44:07.418607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.418541 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\"" Nov 01 21:44:07.418662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.418613 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.418749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.418732 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\"" Nov 01 21:44:07.418798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.418770 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 21:44:07.419039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419019 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.419172 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419153 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 21:44:07.419230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419169 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 21:44:07.419230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419195 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.419331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419255 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.419331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419281 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 21:44:07.419584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419559 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\"" Nov 01 21:44:07.419652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419627 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.419721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419671 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjhd\" (UniqueName: \"kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.419774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419717 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw67\" (UniqueName: \"kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67\") pod \"kas-connection-checker-685895c7c9-6qtm8\" (UID: \"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8\") " pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:44:07.419774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419756 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmxc\" (UniqueName: \"kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc\") pod \"kas-connection-checker-685895c7c9-chf9m\" (UID: \"cef8eaf9-2d56-4d3f-a171-3583becf435e\") " pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:44:07.419857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfq5p\" (UniqueName: \"kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p\") pod \"kas-connection-checker-685895c7c9-lfwr9\" (UID: \"cd265d9d-2558-4f1c-8b67-545bf442aa7c\") " pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:44:07.419857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419844 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.419955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.419955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.420046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419963 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmccv\" (UniqueName: \"kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.420114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420041 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\"" Nov 01 21:44:07.420114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420052 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 21:44:07.420207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.419027 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\"" Nov 01 21:44:07.420207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420191 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 21:44:07.420291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420260 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 21:44:07.420473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420432 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 21:44:07.420536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420483 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 21:44:07.420536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420499 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 21:44:07.420634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420608 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.420778 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420760 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.420844 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.420827 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 21:44:07.421219 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421197 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pn6l\"" Nov 01 21:44:07.421292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421222 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.421353 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421301 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/833deb27-8ec7-4e48-9827-a627d0c7fd5c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.421404 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421360 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bzsh4\"" Nov 01 21:44:07.421500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421475 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/833deb27-8ec7-4e48-9827-a627d0c7fd5c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.421585 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421556 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 21:44:07.421726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421709 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.421842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421819 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.422004 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421942 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.422004 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.421999 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 21:44:07.425435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.425418 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 21:44:07.425726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.425711 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 21:44:07.425996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.425982 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pth5p\"" Nov 01 21:44:07.426289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.426276 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.426603 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.426590 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.427010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.426996 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-46fjd\"" Nov 01 21:44:07.428148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.428134 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fzwwl\"" Nov 01 21:44:07.428294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.428281 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 21:44:07.428343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.428295 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 21:44:07.428343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.428301 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 21:44:07.428343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.428319 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 21:44:07.431656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.431627 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmxc\" (UniqueName: \"kubernetes.io/projected/cef8eaf9-2d56-4d3f-a171-3583becf435e-kube-api-access-zbmxc\") pod \"kas-connection-checker-685895c7c9-chf9m\" (UID: \"cef8eaf9-2d56-4d3f-a171-3583becf435e\") " pod="kube-system/kas-connection-checker-685895c7c9-chf9m" Nov 01 21:44:07.431656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.431650 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw67\" (UniqueName: \"kubernetes.io/projected/3665c82e-f3a8-461e-bb7a-d4cb6d6111a8-kube-api-access-sfw67\") pod \"kas-connection-checker-685895c7c9-6qtm8\" (UID: \"3665c82e-f3a8-461e-bb7a-d4cb6d6111a8\") " pod="kube-system/kas-connection-checker-685895c7c9-6qtm8" Nov 01 21:44:07.431804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.431783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfq5p\" (UniqueName: \"kubernetes.io/projected/cd265d9d-2558-4f1c-8b67-545bf442aa7c-kube-api-access-kfq5p\") pod \"kas-connection-checker-685895c7c9-lfwr9\" (UID: \"cd265d9d-2558-4f1c-8b67-545bf442aa7c\") " pod="kube-system/kas-connection-checker-685895c7c9-lfwr9" Nov 01 21:44:07.431851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.431783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjhd\" (UniqueName: \"kubernetes.io/projected/833deb27-8ec7-4e48-9827-a627d0c7fd5c-kube-api-access-mkjhd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7\" (UID: \"833deb27-8ec7-4e48-9827-a627d0c7fd5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" Nov 01 21:44:07.432333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.432314 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmccv\" (UniqueName: \"kubernetes.io/projected/7a4912cf-de1f-4070-9635-47de5fa95ad5-kube-api-access-hmccv\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.493081 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.493063 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:07.493264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.493246 5036 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:07.493344 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.493282 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:07.501545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.501528 5036 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 21:44:07.503331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.503317 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:07.503387 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.503351 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:07.506365 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.506345 5036 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:07.506554 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.506534 5036 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.us-east-2.compute.internal" Nov 01 21:44:07.520468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.520537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520472 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwts\" (UniqueName: \"kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:44:07.520537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520495 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdbf\" (UniqueName: \"kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.520613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.520613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520569 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.520676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520645 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-snapshots\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.520710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.520754 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.520754 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.520849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520772 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.520849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520817 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.520849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520815 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.520849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.521035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:44:07.521035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520880 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.521035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.520992 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.521035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521026 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.521254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb4e811-66fe-409f-8a75-de7e9e9f936f-konnectivity-ca\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:44:07.521254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521148 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.521254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521171 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342e82fa-1f59-4454-beb0-d9807dc3d674-config-volume\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.521254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521187 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.521254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521220 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521255 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f3b2036-627e-4704-91fb-87df2b35c328-service-ca-bundle\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521262 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521301 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521354 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521378 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.521512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521486 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnjg\" (UniqueName: \"kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4c9c\" (UniqueName: \"kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-stats-auth\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521550 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521553 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521623 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521654 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-serving-cert\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521766 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521816 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.521858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521847 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfpw\" (UniqueName: \"kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a79cdc8f-6b2d-4623-b321-be0088cfad25-config\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521912 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-trusted-ca\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.521994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522016 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522127 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-config\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522142 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522319 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.522358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522403 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2nq\" (UniqueName: \"kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr9h\" (UniqueName: \"kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522438 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb4e811-66fe-409f-8a75-de7e9e9f936f-agent-certs\") pod \"konnectivity-agent-4tvq4\" (UID: \"3cb4e811-66fe-409f-8a75-de7e9e9f936f\") " pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522474 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522522 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522587 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522631 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522670 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522675 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522707 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7vn\" (UniqueName: \"kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522865 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522864 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsn4\" (UniqueName: \"kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522896 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522921 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxrd\" (UniqueName: \"kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd\") pod \"volume-data-source-validator-58c9f5f46d-q2tv6\" (UID: \"490c53c8-ef5b-4296-80b5-32a3204b483f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.522972 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523000 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523036 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523089 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342e82fa-1f59-4454-beb0-d9807dc3d674-tmp-dir\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36c671f8-9973-4f01-a790-c286bb2ee732-iptables-alerter-script\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.523163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523125 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523200 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523230 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523266 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523271 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-key\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523305 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523419 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/521c4405-c6bb-4fd3-bcfb-3cad016376e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523460 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523543 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a79cdc8f-6b2d-4623-b321-be0088cfad25-serving-cert\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523544 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523575 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523622 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-cert\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bef85c8b-3d71-48ea-95f5-9104415811b2-original-pull-secret\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523691 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523717 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523734 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7v9\" (UniqueName: \"kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-env-overrides\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523750 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523775 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpb5\" (UniqueName: \"kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523824 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgk8\" (UniqueName: \"kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8\") pod \"migrator-66df5d496b-66qrn\" (UID: \"91699fd6-b0b6-45c7-8d3c-5175c3a30979\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523874 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523893 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcqf\" (UniqueName: \"kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf\") pod \"network-check-source-6f78d59f99-2k5vn\" (UID: \"fddb723d-930f-4b42-8f63-9780761c0119\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-signing-cabundle\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523911 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.523982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524038 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524058 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524089 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524120 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524148 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524173 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.524232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524251 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d6f94c-7d2b-420b-bea6-0bb929dd6194-tmp-dir\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524256 5036 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524322 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csf7t\" (UniqueName: \"kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524349 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-default-certificate\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524384 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524400 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524457 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524484 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524500 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524598 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-tuned\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-tmp\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524658 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxjq\" (UniqueName: \"kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524685 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524695 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/385fff43-5171-48dd-8530-98bdfb742bec-ovn-node-metrics-cert\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524712 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpx4\" (UniqueName: \"kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524747 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth6h\" (UniqueName: \"kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/385fff43-5171-48dd-8530-98bdfb742bec-ovnkube-script-lib\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524890 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.524919 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525093 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525117 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525147 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skmp\" (UniqueName: \"kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525167 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525185 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525211 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525264 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525266 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-config\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525304 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f3b2036-627e-4704-91fb-87df2b35c328-serving-cert\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525330 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525435 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-data-volume\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525518 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e6c0a65-773b-45c9-b18b-bd582d7a6307-serviceca\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525520 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nbl\" (UniqueName: \"kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525545 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525601 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525616 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-daemon-config\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.525931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525826 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.525990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f3b2036-627e-4704-91fb-87df2b35c328-tmp\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526055 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bz8g\" (UniqueName: \"kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g\") pod \"migrator-66df5d496b-zhwws\" (UID: \"d9ae9063-f7f1-450b-adbf-b21c148dea72\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526155 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rjb\" (UniqueName: \"kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526219 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526272 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526323 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526358 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526416 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cni-binary-copy\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526512 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526580 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526583 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.527641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.526728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a4912cf-de1f-4070-9635-47de5fa95ad5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-bltxj\" (UID: \"7a4912cf-de1f-4070-9635-47de5fa95ad5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.531296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.531274 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdbf\" (UniqueName: \"kubernetes.io/projected/cfaa13d7-7690-4203-842a-94b26ba0331b-kube-api-access-8bdbf\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.531358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.531287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwts\" (UniqueName: \"kubernetes.io/projected/37cb1d59-4690-49c0-95d5-ecf9ce11ff7d-kube-api-access-6rwts\") pod \"ingress-canary-wbmm4\" (UID: \"37cb1d59-4690-49c0-95d5-ecf9ce11ff7d\") " pod="openshift-ingress-canary/ingress-canary-wbmm4" Nov 01 21:44:07.531358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.531338 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.532390 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.532369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnjg\" (UniqueName: \"kubernetes.io/projected/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-kube-api-access-wrnjg\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.532610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.532588 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4c9c\" (UniqueName: \"kubernetes.io/projected/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-kube-api-access-w4c9c\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.532725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.532599 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.532813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.532798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2nq\" (UniqueName: \"kubernetes.io/projected/a79cdc8f-6b2d-4623-b321-be0088cfad25-kube-api-access-ml2nq\") pod \"service-ca-operator-544d749c8c-tpjp6\" (UID: \"a79cdc8f-6b2d-4623-b321-be0088cfad25\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536668 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgk8\" (UniqueName: \"kubernetes.io/projected/91699fd6-b0b6-45c7-8d3c-5175c3a30979-kube-api-access-ltgk8\") pod \"migrator-66df5d496b-66qrn\" (UID: \"91699fd6-b0b6-45c7-8d3c-5175c3a30979\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-66qrn" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfpw\" (UniqueName: \"kubernetes.io/projected/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kube-api-access-nhfpw\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536816 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpb5\" (UniqueName: \"kubernetes.io/projected/510010af-f8ad-4b48-9f73-e9ad1b0a2de9-kube-api-access-rdpb5\") pod \"console-operator-7cd56b8dbc-nscn6\" (UID: \"510010af-f8ad-4b48-9f73-e9ad1b0a2de9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcqf\" (UniqueName: \"kubernetes.io/projected/fddb723d-930f-4b42-8f63-9780761c0119-kube-api-access-gzcqf\") pod \"network-check-source-6f78d59f99-2k5vn\" (UID: \"fddb723d-930f-4b42-8f63-9780761c0119\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2k5vn" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.536941 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7v9\" (UniqueName: \"kubernetes.io/projected/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-kube-api-access-gc7v9\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537227 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpx4\" (UniqueName: \"kubernetes.io/projected/2f3b2036-627e-4704-91fb-87df2b35c328-kube-api-access-xrpx4\") pod \"insights-operator-7c4986d449-lqtc9\" (UID: \"2f3b2036-627e-4704-91fb-87df2b35c328\") " pod="openshift-insights/insights-operator-7c4986d449-lqtc9" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537260 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxjq\" (UniqueName: \"kubernetes.io/projected/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-kube-api-access-lwxjq\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.537341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr9h\" (UniqueName: \"kubernetes.io/projected/36c671f8-9973-4f01-a790-c286bb2ee732-kube-api-access-mxr9h\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.537906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537377 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7vn\" (UniqueName: \"kubernetes.io/projected/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-api-access-bn7vn\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.537906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxrd\" (UniqueName: \"kubernetes.io/projected/490c53c8-ef5b-4296-80b5-32a3204b483f-kube-api-access-7wxrd\") pod \"volume-data-source-validator-58c9f5f46d-q2tv6\" (UID: \"490c53c8-ef5b-4296-80b5-32a3204b483f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2tv6" Nov 01 21:44:07.537906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537375 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsn4\" (UniqueName: \"kubernetes.io/projected/0693fb2b-eba1-4b24-a44b-1a98ddfedd54-kube-api-access-9rsn4\") pod \"service-ca-5fffc97bdf-6w2xf\" (UID: \"0693fb2b-eba1-4b24-a44b-1a98ddfedd54\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6w2xf" Nov 01 21:44:07.537906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.537785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csf7t\" (UniqueName: \"kubernetes.io/projected/bd85b460-2a23-473f-98c5-757f7dc2b421-kube-api-access-csf7t\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.538145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.538127 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49cn\" (UniqueName: \"kubernetes.io/projected/a47e4a36-2064-44c4-bf33-1c26e5b7603c-kube-api-access-r49cn\") pod \"network-check-target-t7lmc\" (UID: \"a47e4a36-2064-44c4-bf33-1c26e5b7603c\") " pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:44:07.540787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.540770 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rjb\" (UniqueName: \"kubernetes.io/projected/89d6f94c-7d2b-420b-bea6-0bb929dd6194-kube-api-access-j8rjb\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542085 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skmp\" (UniqueName: \"kubernetes.io/projected/8e6c0a65-773b-45c9-b18b-bd582d7a6307-kube-api-access-9skmp\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth6h\" (UniqueName: \"kubernetes.io/projected/342e82fa-1f59-4454-beb0-d9807dc3d674-kube-api-access-lth6h\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542568 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nbl\" (UniqueName: \"kubernetes.io/projected/385fff43-5171-48dd-8530-98bdfb742bec-kube-api-access-s9nbl\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542588 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bz8g\" (UniqueName: \"kubernetes.io/projected/d9ae9063-f7f1-450b-adbf-b21c148dea72-kube-api-access-6bz8g\") pod \"migrator-66df5d496b-zhwws\" (UID: \"d9ae9063-f7f1-450b-adbf-b21c148dea72\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zhwws" Nov 01 21:44:07.545249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.542821 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"periodic-gathering-h4t2h-kzgn5\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:44:07.627096 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627071 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627118 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627136 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.627200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-conf-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627201 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-socket-dir-parent\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627253 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627268 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627284 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627305 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.627348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627317 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-run\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627367 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627387 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-modprobe-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627381 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627400 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627435 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89d6f94c-7d2b-420b-bea6-0bb929dd6194-hosts-file\") pod \"node-resolver-jxl58\" (UID: \"89d6f94c-7d2b-420b-bea6-0bb929dd6194\") " pod="openshift-dns/node-resolver-jxl58" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627438 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627497 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-netns\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.627497 5036 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627563 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-crio-socket\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.627578 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.12756016 +0000 UTC m=+1.820540644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627598 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627628 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627594 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-netns\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627651 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-socket-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.627668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627653 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627690 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627700 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-dbus\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627689 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-hostroot\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627745 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-var-lib-kubelet\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627753 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627774 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-var-lib-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-node-log\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627809 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627825 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-kubelet\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627832 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-multus-certs\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627843 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-host\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627855 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627876 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627886 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-etc-kubernetes\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627912 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627963 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627969 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-device-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627988 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.627988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36c671f8-9973-4f01-a790-c286bb2ee732-host-slash\") pod \"iptables-alerter-z9t47\" (UID: \"36c671f8-9973-4f01-a790-c286bb2ee732\") " pod="openshift-network-operator/iptables-alerter-z9t47" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.627998 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.127985437 +0000 UTC m=+1.820965922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628030 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628049 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-multus-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628061 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.12805075 +0000 UTC m=+1.821031214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628027 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628051 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-run-k8s-cni-cncf-io\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628119 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628137 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628206 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e6c0a65-773b-45c9-b18b-bd582d7a6307-host\") pod \"node-ca-q2cpx\" (UID: \"8e6c0a65-773b-45c9-b18b-bd582d7a6307\") " pod="openshift-image-registry/node-ca-q2cpx" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628234 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-sys-fs\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628235 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628263 5036 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628267 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628294 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628263 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-ovn\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628314 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.128299975 +0000 UTC m=+1.821280459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bef85c8b-3d71-48ea-95f5-9104415811b2-kubelet-config\") pod \"global-pull-secret-syncer-wr8bc\" (UID: \"bef85c8b-3d71-48ea-95f5-9104415811b2\") " pod="kube-system/global-pull-secret-syncer-wr8bc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628340 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-kubernetes\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628347 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628371 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628396 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.628380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628424 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-bin\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628461 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-d\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628506 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-system-cni-dir\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628501 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-systemd\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628539 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628564 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-log-socket\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628520 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-etc-openvswitch\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628598 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-kubelet\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628608 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628632 5036 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628634 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628658 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-systemd-units\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628668 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.128656556 +0000 UTC m=+1.821637045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628632 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628693 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-cni-netd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628703 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.128693764 +0000 UTC m=+1.821674231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628707 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-sys\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628727 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628742 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628763 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628768 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb0e1e5-b593-4c2b-9950-ae9c967710e1-cnibin\") pod \"multus-additional-cni-plugins-4tw88\" (UID: \"3eb0e1e5-b593-4c2b-9950-ae9c967710e1\") " pod="openshift-multus/multus-additional-cni-plugins-4tw88" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628777 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628802 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-system-cni-dir\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-slash\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628843 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-cnibin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628870 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628881 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628887 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628890 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628905 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.628925 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.12891455 +0000 UTC m=+1.821895035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628942 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628953 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysconfig\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628981 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.628989 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-bin\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629008 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629034 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629029 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-lib-modules\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629052 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e632f276-6eef-4478-8c2c-1f7e4b98a6d7-etc-sysctl-conf\") pod \"tuned-8n2mc\" (UID: \"e632f276-6eef-4478-8c2c-1f7e4b98a6d7\") " pod="openshift-cluster-node-tuning-operator/tuned-8n2mc" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b5cff7d-1650-41de-9984-fd5cbd4b3fea-registration-dir\") pod \"aws-ebs-csi-driver-node-zbvcm\" (UID: \"0b5cff7d-1650-41de-9984-fd5cbd4b3fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbvcm" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629112 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/403cfda3-7bce-4dfa-bb72-b7f9f26568d0-host-var-lib-cni-multus\") pod \"multus-ckgds\" (UID: \"403cfda3-7bce-4dfa-bb72-b7f9f26568d0\") " pod="openshift-multus/multus-ckgds" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.629130 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.629142 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:07.629167 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:08.129159244 +0000 UTC m=+1.822139708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/385fff43-5171-48dd-8530-98bdfb742bec-run-systemd\") pod \"ovnkube-node-lcz57\" (UID: \"385fff43-5171-48dd-8530-98bdfb742bec\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.629209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:07.630421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.630039 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfaa13d7-7690-4203-842a-94b26ba0331b-metrics-certs\") pod \"network-metrics-daemon-vlnwn\" (UID: \"cfaa13d7-7690-4203-842a-94b26ba0331b\") " pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.683298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.683215 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:07.699269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.699245 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" Nov 01 21:44:07.702263 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.702246 5036 scope.go:122] "RemoveContainer" containerID="b8d37f753ca588f396edbdb980fd2007029053b1cae455a42286cef611231b7a" Nov 01 21:44:07.712215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.712198 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vlnwn" Nov 01 21:44:07.881570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.881537 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:07.999400 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:07.999322 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj"] Nov 01 21:44:08.038466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.038420 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vlnwn"] Nov 01 21:44:08.041782 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:08.041752 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa13d7_7690_4203_842a_94b26ba0331b.slice/crio-dd9b1bf49e17c21ed7b0985a0da1617f3956867e2af25864ea61bbe08e7811ae WatchSource:0}: Error finding container dd9b1bf49e17c21ed7b0985a0da1617f3956867e2af25864ea61bbe08e7811ae: Status 404 returned error can't find the container with id dd9b1bf49e17c21ed7b0985a0da1617f3956867e2af25864ea61bbe08e7811ae Nov 01 21:44:08.044011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.043993 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:44:08.081539 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.081507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:08.133635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.133609 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:08.133799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.133650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:08.133799 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133773 5036 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:08.133799 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133783 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133804 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.133806 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133823 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.13380775 +0000 UTC m=+2.826788236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133859 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.133841307 +0000 UTC m=+2.826821785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.133900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:08.133961 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133910 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.133898081 +0000 UTC m=+2.826878544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.133966 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.134012 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.134030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.133970 5036 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.134058 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134074 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.134061822 +0000 UTC m=+2.827042301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : secret "router-metrics-certs-default" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134102 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134110 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134030 5036 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134133 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.134125962 +0000 UTC m=+2.827106426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134168 5036 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134174 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.13416055 +0000 UTC m=+2.827141026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134187 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls podName:342e82fa-1f59-4454-beb0-d9807dc3d674 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.13418153 +0000 UTC m=+2.827161994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls") pod "dns-default-55x22" (UID: "342e82fa-1f59-4454-beb0-d9807dc3d674") : secret "dns-default-metrics-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134219 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:08.134250 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:08.134238 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:09.13423244 +0000 UTC m=+2.827212903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:08.280856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.280826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:08.422183 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.422151 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:08.434987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.434966 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4tvq4" Nov 01 21:44:08.501538 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.501508 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" event={"ID":"7a4912cf-de1f-4070-9635-47de5fa95ad5","Type":"ContainerStarted","Data":"1806970f836cd47af01536478bec2414910eeec5fe4f8f5224e81eec55304905"} Nov 01 21:44:08.503311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.503295 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:44:08.503406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.503389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" event={"ID":"510010af-f8ad-4b48-9f73-e9ad1b0a2de9","Type":"ContainerStarted","Data":"9a44fd5530f18fc30dbbaa3ec13578b428c484d9bca1e64676108f247c4f0f8e"} Nov 01 21:44:08.504591 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.504564 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlnwn" event={"ID":"cfaa13d7-7690-4203-842a-94b26ba0331b","Type":"ContainerStarted","Data":"dd9b1bf49e17c21ed7b0985a0da1617f3956867e2af25864ea61bbe08e7811ae"} Nov 01 21:44:08.552233 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.552168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:08.675769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:08.675737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.030332 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.030298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.144478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144437 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:09.144619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:09.144619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:09.144619 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144597 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:09.144619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144608 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:09.144619 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144619 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144637 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144660 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144677 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144685 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.144662006 +0000 UTC m=+4.837642485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144727 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.144708562 +0000 UTC m=+4.837689056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:09.144877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.144842 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144927 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144956 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.144947573 +0000 UTC m=+4.837928053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.144999 5036 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.145019 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert podName:521c4405-c6bb-4fd3-bcfb-3cad016376e2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.145013483 +0000 UTC m=+4.837993960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5fkz5" (UID: "521c4405-c6bb-4fd3-bcfb-3cad016376e2") : secret "networking-console-plugin-cert" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.145272 5036 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.145321 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls podName:30fdcebb-da92-461b-a1a1-1ab5e46ec0ab nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.145307849 +0000 UTC m=+4.838288327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gtrkf" (UID: "30fdcebb-da92-461b-a1a1-1ab5e46ec0ab") : secret "cluster-monitoring-operator-tls" not found Nov 01 21:44:09.145536 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:09.145382 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:11.145365573 +0000 UTC m=+4.838346058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:44:09.148154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.147779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd85b460-2a23-473f-98c5-757f7dc2b421-metrics-certs\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:09.148154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.148117 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342e82fa-1f59-4454-beb0-d9807dc3d674-metrics-tls\") pod \"dns-default-55x22\" (UID: \"342e82fa-1f59-4454-beb0-d9807dc3d674\") " pod="openshift-dns/dns-default-55x22" Nov 01 21:44:09.157431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.157354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.216413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.216380 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55x22" Nov 01 21:44:09.284318 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.284254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.490580 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.490544 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.501271 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.501244 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55x22"] Nov 01 21:44:09.510978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.510950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlnwn" event={"ID":"cfaa13d7-7690-4203-842a-94b26ba0331b","Type":"ContainerStarted","Data":"21ec5c9659c1f84f25226d8487d1ed82fefa01fe3bed758443c387eced623c92"} Nov 01 21:44:09.511079 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.511007 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vlnwn" event={"ID":"cfaa13d7-7690-4203-842a-94b26ba0331b","Type":"ContainerStarted","Data":"95fc3007f9a9e5da81d9f8f4220ebbb97e13c6dacc73322ede9d9e3c52218d7b"} Nov 01 21:44:09.512266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.512242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55x22" event={"ID":"342e82fa-1f59-4454-beb0-d9807dc3d674","Type":"ContainerStarted","Data":"9ff9f82bdb6b77b7411526b5b8144b1ab299721825386598fa255debe46d8b75"} Nov 01 21:44:09.680612 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.680501 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:09.891606 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:09.891570 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:10.021840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.021807 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:10.222274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.222198 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:10.348750 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.348708 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:10.518698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.518663 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" event={"ID":"7a4912cf-de1f-4070-9635-47de5fa95ad5","Type":"ContainerStarted","Data":"c00b284c92d05c50f24a7686ab0d8fb2c2cf52e41c1208268884cb61278eb52c"} Nov 01 21:44:10.519075 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.518724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-bltxj" event={"ID":"7a4912cf-de1f-4070-9635-47de5fa95ad5","Type":"ContainerStarted","Data":"be1fdb649b0e0b1dfd93c27821d81e46de7cc93f0992545d59e789e61012f7f6"} Nov 01 21:44:10.607190 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:10.607156 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.165063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:11.165259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165075 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:11.165259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165117 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:11.165259 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165213 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:11.165259 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165232 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c45c69888-j7vl7: secret "image-registry-tls" not found Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165296 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls podName:aba3b56e-3966-4258-8169-b20577969634 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:15.165277206 +0000 UTC m=+8.858257685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls") pod "image-registry-6c45c69888-j7vl7" (UID: "aba3b56e-3966-4258-8169-b20577969634") : secret "image-registry-tls" not found Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165305 5036 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165324 5036 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-596f45d76f-hl6hx: secret "image-registry-tls" not found Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165322 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165380 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls podName:73ec954c-cf47-42ac-86f8-0daab114e5f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:15.165366798 +0000 UTC m=+8.858347275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls") pod "image-registry-596f45d76f-hl6hx" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2") : secret "image-registry-tls" not found Nov 01 21:44:11.165417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:11.165647 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165434 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle podName:bd85b460-2a23-473f-98c5-757f7dc2b421 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:15.16542268 +0000 UTC m=+8.858403151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle") pod "router-default-554f8bcb97-6mq5w" (UID: "bd85b460-2a23-473f-98c5-757f7dc2b421") : configmap references non-existent config key: service-ca.crt Nov 01 21:44:11.165647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.165517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:11.165647 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165605 5036 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 21:44:11.165647 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:11.165637 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls podName:6777c192-94cb-4df3-8ed7-3ce7688c0ba7 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:15.165629956 +0000 UTC m=+8.858610420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4bjns" (UID: "6777c192-94cb-4df3-8ed7-3ce7688c0ba7") : secret "insights-runtime-extractor-tls" not found Nov 01 21:44:11.167744 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.167718 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/30fdcebb-da92-461b-a1a1-1ab5e46ec0ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gtrkf\" (UID: \"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:11.167889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.167752 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/521c4405-c6bb-4fd3-bcfb-3cad016376e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5fkz5\" (UID: \"521c4405-c6bb-4fd3-bcfb-3cad016376e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:11.198987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.198959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.302920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.302891 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" Nov 01 21:44:11.309900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.309883 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" Nov 01 21:44:11.325747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.325717 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.366289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.366254 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:11.469015 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.468986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.524452 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.524413 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55x22" event={"ID":"342e82fa-1f59-4454-beb0-d9807dc3d674","Type":"ContainerStarted","Data":"ffb56e27b052561bb1efda344e8c8b427ee9108a37d8bff362d97448fc8caabb"} Nov 01 21:44:11.524790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.524493 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55x22" event={"ID":"342e82fa-1f59-4454-beb0-d9807dc3d674","Type":"ContainerStarted","Data":"9358a2e16c2fc3ad1c3209df8d2437a0d7a49dae5e6abe7282137055121a7c42"} Nov 01 21:44:11.591754 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.591721 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.599164 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.599143 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf"] Nov 01 21:44:11.602119 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:11.602094 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fdcebb_da92_461b_a1a1_1ab5e46ec0ab.slice/crio-9206fe07ac434ad1e9be745526fbc7c0cd7bcc39388ca9e5fc1e57093935d45e WatchSource:0}: Error finding container 9206fe07ac434ad1e9be745526fbc7c0cd7bcc39388ca9e5fc1e57093935d45e: Status 404 returned error can't find the container with id 9206fe07ac434ad1e9be745526fbc7c0cd7bcc39388ca9e5fc1e57093935d45e Nov 01 21:44:11.604022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.604002 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5fkz5"] Nov 01 21:44:11.607162 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:11.607140 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521c4405_c6bb_4fd3_bcfb_3cad016376e2.slice/crio-f1e24a90c25f575a566d095ce9715e569ec810c5a6a7da008ea74077fb668a1a WatchSource:0}: Error finding container f1e24a90c25f575a566d095ce9715e569ec810c5a6a7da008ea74077fb668a1a: Status 404 returned error can't find the container with id f1e24a90c25f575a566d095ce9715e569ec810c5a6a7da008ea74077fb668a1a Nov 01 21:44:11.717375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.717304 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:11.838932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.838908 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-55x22" Nov 01 21:44:11.878818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:11.878784 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:12.085624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:12.085592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:12.232848 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:12.232819 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:44:12.284518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:12.284430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:12.530331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:12.530296 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" event={"ID":"521c4405-c6bb-4fd3-bcfb-3cad016376e2","Type":"ContainerStarted","Data":"f1e24a90c25f575a566d095ce9715e569ec810c5a6a7da008ea74077fb668a1a"} Nov 01 21:44:12.531867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:12.531831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" event={"ID":"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab","Type":"ContainerStarted","Data":"9206fe07ac434ad1e9be745526fbc7c0cd7bcc39388ca9e5fc1e57093935d45e"} Nov 01 21:44:13.360642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.360561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:13.489640 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.489618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:13.537649 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.537621 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gtrkf" event={"ID":"30fdcebb-da92-461b-a1a1-1ab5e46ec0ab","Type":"ContainerStarted","Data":"68601fe20fad6c9bab336375343ebf207cccc3720e780831e2e4ab75dc25ca5f"} Nov 01 21:44:13.539333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.539308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5fkz5" event={"ID":"521c4405-c6bb-4fd3-bcfb-3cad016376e2","Type":"ContainerStarted","Data":"593e69d71f4f5650b16ecb81c292c7138921ea2d243f1e229fb09c1e140c6b0e"} Nov 01 21:44:13.557506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.557480 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:13.557603 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.557586 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:13.561764 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.561744 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nscn6" Nov 01 21:44:13.684916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.684839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:13.813973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.813932 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:13.938746 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:13.938666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:14.069798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:14.069767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:14.194093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:14.194020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:14.329421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:14.329374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:14.889094 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:14.889063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:15.079834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.079787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:15.201432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.201352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:15.201432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.201421 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:15.201680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.201491 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:15.201680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.201534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:15.202017 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.201991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd85b460-2a23-473f-98c5-757f7dc2b421-service-ca-bundle\") pod \"router-default-554f8bcb97-6mq5w\" (UID: \"bd85b460-2a23-473f-98c5-757f7dc2b421\") " pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:15.203738 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.203710 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6777c192-94cb-4df3-8ed7-3ce7688c0ba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bjns\" (UID: \"6777c192-94cb-4df3-8ed7-3ce7688c0ba7\") " pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:15.203852 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.203828 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"image-registry-6c45c69888-j7vl7\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:15.203929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.203921 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:15.203984 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.203947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"image-registry-596f45d76f-hl6hx\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:15.215834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.215816 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:15.222918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.222765 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bjns" Nov 01 21:44:15.485932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.485737 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:44:15.488755 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:15.488728 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3b56e_3966_4258_8169_b20577969634.slice/crio-f879b11883038181fe16a67afe89d8755ee75417934c0be7c2eed4ded06f3520 WatchSource:0}: Error finding container f879b11883038181fe16a67afe89d8755ee75417934c0be7c2eed4ded06f3520: Status 404 returned error can't find the container with id f879b11883038181fe16a67afe89d8755ee75417934c0be7c2eed4ded06f3520 Nov 01 21:44:15.500458 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.500425 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:15.512120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.512012 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bjns"] Nov 01 21:44:15.515245 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:15.515218 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6777c192_94cb_4df3_8ed7_3ce7688c0ba7.slice/crio-fd939f1d218d4bd3a73d496887232d41d7d88ad54f6c4942078dad561595f408 WatchSource:0}: Error finding container fd939f1d218d4bd3a73d496887232d41d7d88ad54f6c4942078dad561595f408: Status 404 returned error can't find the container with id fd939f1d218d4bd3a73d496887232d41d7d88ad54f6c4942078dad561595f408 Nov 01 21:44:15.516036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.516014 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:44:15.518404 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:15.518383 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ec954c_cf47_42ac_86f8_0daab114e5f2.slice/crio-c631a065edd6bced98e7628f58dd9e508f845b47317b5c3d21f25391e508c8ec WatchSource:0}: Error finding container c631a065edd6bced98e7628f58dd9e508f845b47317b5c3d21f25391e508c8ec: Status 404 returned error can't find the container with id c631a065edd6bced98e7628f58dd9e508f845b47317b5c3d21f25391e508c8ec Nov 01 21:44:15.569352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.568901 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" event={"ID":"aba3b56e-3966-4258-8169-b20577969634","Type":"ContainerStarted","Data":"f879b11883038181fe16a67afe89d8755ee75417934c0be7c2eed4ded06f3520"} Nov 01 21:44:15.581741 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.572634 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bjns" event={"ID":"6777c192-94cb-4df3-8ed7-3ce7688c0ba7","Type":"ContainerStarted","Data":"fd939f1d218d4bd3a73d496887232d41d7d88ad54f6c4942078dad561595f408"} Nov 01 21:44:15.583472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.583424 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" event={"ID":"73ec954c-cf47-42ac-86f8-0daab114e5f2","Type":"ContainerStarted","Data":"c631a065edd6bced98e7628f58dd9e508f845b47317b5c3d21f25391e508c8ec"} Nov 01 21:44:15.793977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:15.793941 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-554f8bcb97-6mq5w"] Nov 01 21:44:15.796830 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:15.796788 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd85b460_2a23_473f_98c5_757f7dc2b421.slice/crio-ca9f6fda7df56c6f457cd2371e54843898ec09ac4812f89b19463c316efb3ad6 WatchSource:0}: Error finding container ca9f6fda7df56c6f457cd2371e54843898ec09ac4812f89b19463c316efb3ad6: Status 404 returned error can't find the container with id ca9f6fda7df56c6f457cd2371e54843898ec09ac4812f89b19463c316efb3ad6 Nov 01 21:44:16.588935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.588904 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" event={"ID":"aba3b56e-3966-4258-8169-b20577969634","Type":"ContainerStarted","Data":"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f"} Nov 01 21:44:16.590890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.590869 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bjns" event={"ID":"6777c192-94cb-4df3-8ed7-3ce7688c0ba7","Type":"ContainerStarted","Data":"33f460e4831de089f3227223261c80e6de99e5822c9fb3c46467d0574de89fe1"} Nov 01 21:44:16.591001 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.590917 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bjns" event={"ID":"6777c192-94cb-4df3-8ed7-3ce7688c0ba7","Type":"ContainerStarted","Data":"cd1386fa0f4b0a71d4ca71fb243f4632a8cfcaa302c1dd5d7c591fa7aea01f00"} Nov 01 21:44:16.592420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.592355 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" event={"ID":"73ec954c-cf47-42ac-86f8-0daab114e5f2","Type":"ContainerStarted","Data":"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a"} Nov 01 21:44:16.593817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.593795 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" event={"ID":"bd85b460-2a23-473f-98c5-757f7dc2b421","Type":"ContainerStarted","Data":"c8f0fedb8a53ecc88f52a6df99336360fdf41211b1d8d6bc74ea6ac7f2d9a932"} Nov 01 21:44:16.593928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.593837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" event={"ID":"bd85b460-2a23-473f-98c5-757f7dc2b421","Type":"ContainerStarted","Data":"ca9f6fda7df56c6f457cd2371e54843898ec09ac4812f89b19463c316efb3ad6"} Nov 01 21:44:16.744745 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.744671 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:16.744936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.744921 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:16.913585 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:16.913558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcz57" Nov 01 21:44:17.501029 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:17.500996 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:17.504014 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:17.503990 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:17.551041 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:17.551011 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:17.598268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:17.598233 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:18.604040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:18.604004 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bjns" event={"ID":"6777c192-94cb-4df3-8ed7-3ce7688c0ba7","Type":"ContainerStarted","Data":"149cfdaa77db8dccc5daadd65b999a3998d48e3474bd0700d2ddfd5e48dac4b1"} Nov 01 21:44:18.604040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:18.604040 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:18.856329 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:18.856233 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:44:20.054806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:20.054776 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:22.542129 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:22.542097 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-55x22" Nov 01 21:44:22.660822 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:22.660791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:22.785666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:22.785631 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:23.108066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.108026 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:44:23.220477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.220426 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:23.348919 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.348891 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:23.349069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.349025 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:23.350620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.350600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-554f8bcb97-6mq5w" Nov 01 21:44:23.381394 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:23.381322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:26.201300 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:26.201262 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:26.329623 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:26.329588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:27.066651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:27.066611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:27.193724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:27.193679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:29.631157 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:29.631126 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:30.273929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:30.273898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:30.408374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:30.408340 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:30.941713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:30.941666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:31.066103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:31.066040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.230648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.230618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.235675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.235650 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7lmc" Nov 01 21:44:32.366665 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.366622 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.438234 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.438165 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cktms"] Nov 01 21:44:32.471105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.471079 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.501726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.501695 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.522551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.522528 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 21:44:32.522551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.522531 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 21:44:32.522707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.522618 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bpkb5\"" Nov 01 21:44:32.522932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.522903 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 21:44:32.522932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.522909 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 21:44:32.628471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.628438 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.637410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637380 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-sys\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637434 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-accelerators-collector-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637512 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6hg\" (UniqueName: \"kubernetes.io/projected/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-kube-api-access-7f6hg\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-metrics-client-ca\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637644 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-wtmp\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637675 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-textfile\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637743 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637771 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-tls\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.637875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.637857 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-root\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738678 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-tls\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-root\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738778 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-sys\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-accelerators-collector-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6hg\" (UniqueName: \"kubernetes.io/projected/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-kube-api-access-7f6hg\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-metrics-client-ca\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.738936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738923 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-wtmp\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.739181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.738967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-textfile\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.739181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.739009 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.739389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.739360 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-root\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.740091 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.740066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-metrics-client-ca\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.740188 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.740144 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-accelerators-collector-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.740253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.740215 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-sys\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.740351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.740334 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-wtmp\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.740616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.740584 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-textfile\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.742271 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.742245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.742467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.742429 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-node-exporter-tls\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.754883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.754853 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6hg\" (UniqueName: \"kubernetes.io/projected/a68e7dfb-ed88-40cd-b1c9-da441de42fa9-kube-api-access-7f6hg\") pod \"node-exporter-cktms\" (UID: \"a68e7dfb-ed88-40cd-b1c9-da441de42fa9\") " pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.757604 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.757574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:32.821253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.821221 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cktms" Nov 01 21:44:32.861094 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:44:32.861069 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68e7dfb_ed88_40cd_b1c9_da441de42fa9.slice/crio-79f1941612652cc986be10aa03bf825b7fbed93c63a65de12aec987edf991a51 WatchSource:0}: Error finding container 79f1941612652cc986be10aa03bf825b7fbed93c63a65de12aec987edf991a51: Status 404 returned error can't find the container with id 79f1941612652cc986be10aa03bf825b7fbed93c63a65de12aec987edf991a51 Nov 01 21:44:32.880863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:32.880839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:33.112396 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:33.112372 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:33.312066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:33.312030 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:33.455383 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:33.455298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:33.583872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:33.583781 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:33.660837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:33.660782 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cktms" event={"ID":"a68e7dfb-ed88-40cd-b1c9-da441de42fa9","Type":"ContainerStarted","Data":"79f1941612652cc986be10aa03bf825b7fbed93c63a65de12aec987edf991a51"} Nov 01 21:44:34.668185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:34.668123 5036 generic.go:356] "Generic (PLEG): container finished" podID="a68e7dfb-ed88-40cd-b1c9-da441de42fa9" containerID="75a3797621386e94271b20370393d95f6c562081c6e2e0efd295885f47d68701" exitCode=0 Nov 01 21:44:34.668185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:34.668172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cktms" event={"ID":"a68e7dfb-ed88-40cd-b1c9-da441de42fa9","Type":"ContainerDied","Data":"75a3797621386e94271b20370393d95f6c562081c6e2e0efd295885f47d68701"} Nov 01 21:44:35.460650 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.460600 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:35.588992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.588956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:35.674424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.674389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cktms" event={"ID":"a68e7dfb-ed88-40cd-b1c9-da441de42fa9","Type":"ContainerStarted","Data":"88bea7cc166e9d8cfa25426fafaf4129cd27ab31830f93182056d9204e486feb"} Nov 01 21:44:35.674855 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.674492 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cktms" event={"ID":"a68e7dfb-ed88-40cd-b1c9-da441de42fa9","Type":"ContainerStarted","Data":"4d8c34401aec5ac82b350551b98aa8747d993d63de98ce3677cea56a0d95b870"} Nov 01 21:44:35.698714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.698662 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cktms" podStartSLOduration=2.110512929 podStartE2EDuration="3.698649027s" totalImagesPullingTime="1.508824447s" totalInitContainerRuntime="79.311651ms" isStatefulPod=true podCreationTimestamp="2026-11-01 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:35.696601251 +0000 UTC m=+29.389581738" watchObservedRunningTime="2026-11-01 21:44:35.698649027 +0000 UTC m=+29.391629490" Nov 01 21:44:35.784803 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.784774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:35.925936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:35.925904 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:36.681735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:36.681702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:36.815659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:36.815625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:37.342098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:37.342060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:37.523482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:37.523425 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:38.609240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:38.609214 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:44:38.682374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:38.682339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:38.823584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:38.823552 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:39.033943 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:39.033906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:39.905186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:39.905154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:40.057461 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:40.057419 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:41.202123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:41.202088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:41.336859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:41.336828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:41.712213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:41.712173 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:41.841502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:41.841468 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:42.611129 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:42.611099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:42.742310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:42.742270 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:43.523136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:43.523099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:43.651746 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:43.651704 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:45.297030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:45.296998 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:44:45.431240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:45.431208 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:45.564919 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:45.564851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:47.462812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:47.462724 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:47.591872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:47.591832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:48.128575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.128540 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" podUID="aba3b56e-3966-4258-8169-b20577969634" containerName="registry" containerID="cri-o://ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f" gracePeriod=30 Nov 01 21:44:48.366165 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.366139 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:48.462758 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.462691 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.462758 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.462736 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.462770 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.462802 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.462975 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463025 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463060 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463106 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") pod \"aba3b56e-3966-4258-8169-b20577969634\" (UID: \"aba3b56e-3966-4258-8169-b20577969634\") " Nov 01 21:44:48.463261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463207 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:44:48.463865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463823 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:44:48.463995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463934 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-trusted-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.463995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.463966 5036 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3b56e-3966-4258-8169-b20577969634-registry-certificates\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.467542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.466425 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:44:48.469769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.469745 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "kube-api-access-9mp6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:44:48.469891 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.469761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:44:48.470338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.470310 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:44:48.471202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.471181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:44:48.475413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.475387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted" pod "aba3b56e-3966-4258-8169-b20577969634" (UID: "aba3b56e-3966-4258-8169-b20577969634"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:44:48.564804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564783 5036 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3b56e-3966-4258-8169-b20577969634-ca-trust-extracted\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.564804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564802 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp6v\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-kube-api-access-9mp6v\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.564925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564814 5036 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-installation-pull-secrets\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.564925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564823 5036 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-bound-sa-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.564925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564833 5036 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3b56e-3966-4258-8169-b20577969634-image-registry-private-configuration\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.564925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.564842 5036 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3b56e-3966-4258-8169-b20577969634-registry-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:44:48.725801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.725736 5036 generic.go:356] "Generic (PLEG): container finished" podID="aba3b56e-3966-4258-8169-b20577969634" containerID="ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f" exitCode=0 Nov 01 21:44:48.725801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.725768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" event={"ID":"aba3b56e-3966-4258-8169-b20577969634","Type":"ContainerDied","Data":"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f"} Nov 01 21:44:48.725934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.725797 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" Nov 01 21:44:48.725934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.725829 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c45c69888-j7vl7" event={"ID":"aba3b56e-3966-4258-8169-b20577969634","Type":"ContainerDied","Data":"f879b11883038181fe16a67afe89d8755ee75417934c0be7c2eed4ded06f3520"} Nov 01 21:44:48.725934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.725848 5036 scope.go:122] "RemoveContainer" containerID="ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f" Nov 01 21:44:48.735205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.735188 5036 scope.go:122] "RemoveContainer" containerID="ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f" Nov 01 21:44:48.735454 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:48.735421 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f\": container with ID starting with ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f not found: ID does not exist" containerID="ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f" Nov 01 21:44:48.735507 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.735471 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f"} err="failed to get container status \"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f\": rpc error: code = NotFound desc = could not find container \"ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f\": container with ID starting with ca8cbb255f92baa855e711d2f5d41ef235eb25bc32658f268f855e814ba5230f not found: ID does not exist" Nov 01 21:44:48.748366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.748343 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:44:48.752373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:48.752351 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c45c69888-j7vl7"] Nov 01 21:44:48.855780 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:44:48.855752 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:44:50.435498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:50.435463 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba3b56e-3966-4258-8169-b20577969634" path="/var/lib/kubelet/pods/aba3b56e-3966-4258-8169-b20577969634/volumes" Nov 01 21:44:53.108693 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:53.108654 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:53.230069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:53.230035 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:55.679809 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:55.679773 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:55.833671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:55.833638 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:56.202985 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:56.202946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:56.425061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:44:56.425029 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:10.321770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.321710 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" podUID="73ec954c-cf47-42ac-86f8-0daab114e5f2" containerName="registry" containerID="cri-o://06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a" gracePeriod=30 Nov 01 21:45:10.573887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.573836 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:45:10.625747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625720 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.625875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625765 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.625875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625791 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.625875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625811 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.625875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.625875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.626145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625895 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.626145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.625942 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") pod \"73ec954c-cf47-42ac-86f8-0daab114e5f2\" (UID: \"73ec954c-cf47-42ac-86f8-0daab114e5f2\") " Nov 01 21:45:10.626438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.626357 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:10.626569 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.626503 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:10.628331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.628306 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:10.628433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.628307 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "kube-api-access-kbsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:10.628546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.628458 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:10.628667 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.628648 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:10.628716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.628668 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:10.635204 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.635181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted" pod "73ec954c-cf47-42ac-86f8-0daab114e5f2" (UID: "73ec954c-cf47-42ac-86f8-0daab114e5f2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:45:10.727064 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727039 5036 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-bound-sa-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727064 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727062 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-trusted-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727072 5036 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-installation-pull-secrets\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727082 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsxl\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-kube-api-access-kbsxl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727092 5036 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-certificates\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727101 5036 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73ec954c-cf47-42ac-86f8-0daab114e5f2-ca-trust-extracted\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727110 5036 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73ec954c-cf47-42ac-86f8-0daab114e5f2-image-registry-private-configuration\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.727180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.727119 5036 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73ec954c-cf47-42ac-86f8-0daab114e5f2-registry-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:10.812820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.812789 5036 generic.go:356] "Generic (PLEG): container finished" podID="73ec954c-cf47-42ac-86f8-0daab114e5f2" containerID="06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a" exitCode=0 Nov 01 21:45:10.812923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.812843 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" Nov 01 21:45:10.812923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.812881 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" event={"ID":"73ec954c-cf47-42ac-86f8-0daab114e5f2","Type":"ContainerDied","Data":"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a"} Nov 01 21:45:10.813043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.812946 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596f45d76f-hl6hx" event={"ID":"73ec954c-cf47-42ac-86f8-0daab114e5f2","Type":"ContainerDied","Data":"c631a065edd6bced98e7628f58dd9e508f845b47317b5c3d21f25391e508c8ec"} Nov 01 21:45:10.813043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.812976 5036 scope.go:122] "RemoveContainer" containerID="06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a" Nov 01 21:45:10.821826 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.821812 5036 scope.go:122] "RemoveContainer" containerID="06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a" Nov 01 21:45:10.822101 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:45:10.822084 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a\": container with ID starting with 06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a not found: ID does not exist" containerID="06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a" Nov 01 21:45:10.822159 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.822107 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a"} err="failed to get container status \"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a\": rpc error: code = NotFound desc = could not find container \"06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a\": container with ID starting with 06f621b9d9ce8ef9ab22a78e999646c7d1e0d51b48465e26d85959e3d264230a not found: ID does not exist" Nov 01 21:45:10.834182 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.834138 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:45:10.837750 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:10.837729 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-596f45d76f-hl6hx"] Nov 01 21:45:11.209153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.209081 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:11.333828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.333798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:11.786961 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.786929 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:11.818624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.818596 5036 generic.go:356] "Generic (PLEG): container finished" podID="833deb27-8ec7-4e48-9827-a627d0c7fd5c" containerID="7117a20130c7d82b73be1034a63b4cc02bf3e9f4d1e8dbb94ef354bf9ff0cbe8" exitCode=0 Nov 01 21:45:11.818769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.818671 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" event={"ID":"833deb27-8ec7-4e48-9827-a627d0c7fd5c","Type":"ContainerDied","Data":"7117a20130c7d82b73be1034a63b4cc02bf3e9f4d1e8dbb94ef354bf9ff0cbe8"} Nov 01 21:45:11.819026 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.819004 5036 scope.go:122] "RemoveContainer" containerID="7117a20130c7d82b73be1034a63b4cc02bf3e9f4d1e8dbb94ef354bf9ff0cbe8" Nov 01 21:45:11.820280 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.820258 5036 generic.go:356] "Generic (PLEG): container finished" podID="2f3b2036-627e-4704-91fb-87df2b35c328" containerID="ecbe335295609b5460bd8e51923d0ad961599a1b51cf4917ee4aefd4382142ca" exitCode=0 Nov 01 21:45:11.820389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.820312 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" event={"ID":"2f3b2036-627e-4704-91fb-87df2b35c328","Type":"ContainerDied","Data":"ecbe335295609b5460bd8e51923d0ad961599a1b51cf4917ee4aefd4382142ca"} Nov 01 21:45:11.820582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.820567 5036 scope.go:122] "RemoveContainer" containerID="ecbe335295609b5460bd8e51923d0ad961599a1b51cf4917ee4aefd4382142ca" Nov 01 21:45:11.983074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:11.982978 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:12.202192 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.202117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:12.272188 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.272158 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:45:12.272382 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.272360 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" podUID="793bff98-0107-4731-a6a0-5747f9303443" containerName="insights-gathering" containerID="cri-o://234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e" gracePeriod=30 Nov 01 21:45:12.343552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.343525 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:12.362321 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362291 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vhpgm-lfdc2"] Nov 01 21:45:12.362679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362660 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aba3b56e-3966-4258-8169-b20577969634" containerName="registry" Nov 01 21:45:12.362770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362681 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba3b56e-3966-4258-8169-b20577969634" containerName="registry" Nov 01 21:45:12.362770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362699 5036 cpu_manager.go:414] "containerMap: removing container" podUID="73ec954c-cf47-42ac-86f8-0daab114e5f2" containerName="registry" Nov 01 21:45:12.362770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362707 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec954c-cf47-42ac-86f8-0daab114e5f2" containerName="registry" Nov 01 21:45:12.362921 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362795 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba3b56e-3966-4258-8169-b20577969634" containerName="registry" Nov 01 21:45:12.362921 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.362809 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec954c-cf47-42ac-86f8-0daab114e5f2" containerName="registry" Nov 01 21:45:12.368006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.367987 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.374566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.374542 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vhpgm-lfdc2"] Nov 01 21:45:12.435763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.435733 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ec954c-cf47-42ac-86f8-0daab114e5f2" path="/var/lib/kubelet/pods/73ec954c-cf47-42ac-86f8-0daab114e5f2/volumes" Nov 01 21:45:12.439776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.439755 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpgq\" (UniqueName: \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.439860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.439809 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.439965 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.439944 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.440019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.439983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.525868 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.525846 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:45:12.541098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.541075 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpgq\" (UniqueName: \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.541207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.541124 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.541207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.541150 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.541325 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.541276 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.543918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.541774 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.543918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.542236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.543918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.542814 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.553497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.553472 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpgq\" (UniqueName: \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\") pod \"periodic-gathering-vhpgm-lfdc2\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.641972 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.641934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") pod \"793bff98-0107-4731-a6a0-5747f9303443\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " Nov 01 21:45:12.642097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642009 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") pod \"793bff98-0107-4731-a6a0-5747f9303443\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " Nov 01 21:45:12.642097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642074 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") pod \"793bff98-0107-4731-a6a0-5747f9303443\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " Nov 01 21:45:12.642214 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642139 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") pod \"793bff98-0107-4731-a6a0-5747f9303443\" (UID: \"793bff98-0107-4731-a6a0-5747f9303443\") " Nov 01 21:45:12.642266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642240 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle" pod "793bff98-0107-4731-a6a0-5747f9303443" (UID: "793bff98-0107-4731-a6a0-5747f9303443"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:12.642378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642360 5036 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-service-ca-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:12.642438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642361 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path" pod "793bff98-0107-4731-a6a0-5747f9303443" (UID: "793bff98-0107-4731-a6a0-5747f9303443"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:45:12.642438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.642378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle" pod "793bff98-0107-4731-a6a0-5747f9303443" (UID: "793bff98-0107-4731-a6a0-5747f9303443"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:12.644209 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.644190 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh" pod "793bff98-0107-4731-a6a0-5747f9303443" (UID: "793bff98-0107-4731-a6a0-5747f9303443"). InnerVolumeSpecName "kube-api-access-2l4dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:12.681278 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.681254 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:45:12.743047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.743019 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/793bff98-0107-4731-a6a0-5747f9303443-trusted-ca-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:12.743185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.743049 5036 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/793bff98-0107-4731-a6a0-5747f9303443-archives-path\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:12.743185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.743062 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4dh\" (UniqueName: \"kubernetes.io/projected/793bff98-0107-4731-a6a0-5747f9303443-kube-api-access-2l4dh\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:12.826944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.826915 5036 generic.go:356] "Generic (PLEG): container finished" podID="793bff98-0107-4731-a6a0-5747f9303443" containerID="234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e" exitCode=2 Nov 01 21:45:12.827095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.826992 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" Nov 01 21:45:12.827095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.826996 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" event={"ID":"793bff98-0107-4731-a6a0-5747f9303443","Type":"ContainerDied","Data":"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e"} Nov 01 21:45:12.827213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.827092 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h4t2h-kzgn5" event={"ID":"793bff98-0107-4731-a6a0-5747f9303443","Type":"ContainerDied","Data":"307de04b58832f23c63c3feb7fe907e9aa5388ce89559c80a23c2df080e40516"} Nov 01 21:45:12.827213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.827123 5036 scope.go:122] "RemoveContainer" containerID="234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e" Nov 01 21:45:12.829236 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.829216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2qbh7" event={"ID":"833deb27-8ec7-4e48-9827-a627d0c7fd5c","Type":"ContainerStarted","Data":"eb77c0c4ca3cbeed0d22e900d7ce153d907e4791c32c4974040366f471e7e73c"} Nov 01 21:45:12.831676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.831656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lqtc9" event={"ID":"2f3b2036-627e-4704-91fb-87df2b35c328","Type":"ContainerStarted","Data":"a7d4d8e8dfb4e41807608236445a2c74a23601a9871260eb23eadca132db3e8b"} Nov 01 21:45:12.835630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.835592 5036 scope.go:122] "RemoveContainer" containerID="234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e" Nov 01 21:45:12.835866 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:45:12.835835 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e\": container with ID starting with 234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e not found: ID does not exist" containerID="234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e" Nov 01 21:45:12.835956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.835863 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e"} err="failed to get container status \"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e\": rpc error: code = NotFound desc = could not find container \"234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e\": container with ID starting with 234130c27697b412b2571154406c4f305784f4a8ecf64b7f7040516216b10c7e not found: ID does not exist" Nov 01 21:45:12.865230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.865191 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:45:12.872940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.872917 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-h4t2h-kzgn5"] Nov 01 21:45:12.957827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:12.957758 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vhpgm-lfdc2"] Nov 01 21:45:12.960244 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:45:12.960215 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b9dd27_1e90_4edc_8646_3e7979ef591a.slice/crio-b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4 WatchSource:0}: Error finding container b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4: Status 404 returned error can't find the container with id b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4 Nov 01 21:45:13.841205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:13.841171 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" event={"ID":"f7b9dd27-1e90-4edc-8646-3e7979ef591a","Type":"ContainerStarted","Data":"f05f7255503027926fc18d4abbef5cbfce3cf8e61f52379d80ffa1d2ebe01f72"} Nov 01 21:45:13.841607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:13.841230 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" event={"ID":"f7b9dd27-1e90-4edc-8646-3e7979ef591a","Type":"ContainerStarted","Data":"b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4"} Nov 01 21:45:13.860301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:13.860259 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" podStartSLOduration=1.860247002 podStartE2EDuration="1.860247002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:45:13.858903068 +0000 UTC m=+67.551883554" watchObservedRunningTime="2026-11-01 21:45:13.860247002 +0000 UTC m=+67.553227488" Nov 01 21:45:14.434994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:14.434960 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793bff98-0107-4731-a6a0-5747f9303443" path="/var/lib/kubelet/pods/793bff98-0107-4731-a6a0-5747f9303443/volumes" Nov 01 21:45:16.936144 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:16.936114 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-554f8bcb97-6mq5w_bd85b460-2a23-473f-98c5-757f7dc2b421/router/0.log" Nov 01 21:45:17.012699 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:17.012672 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wbmm4_37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/serve-healthcheck-canary/0.log" Nov 01 21:45:18.856255 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:45:18.856224 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:45:21.873843 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:21.873811 5036 generic.go:356] "Generic (PLEG): container finished" podID="a79cdc8f-6b2d-4623-b321-be0088cfad25" containerID="5916ddec60564c7ef70909f124c69a68c93cc376218e34ef94f5c4f84609526c" exitCode=0 Nov 01 21:45:21.874222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:21.873849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" event={"ID":"a79cdc8f-6b2d-4623-b321-be0088cfad25","Type":"ContainerDied","Data":"5916ddec60564c7ef70909f124c69a68c93cc376218e34ef94f5c4f84609526c"} Nov 01 21:45:21.874222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:21.874098 5036 scope.go:122] "RemoveContainer" containerID="5916ddec60564c7ef70909f124c69a68c93cc376218e34ef94f5c4f84609526c" Nov 01 21:45:22.601468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:22.598733 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:22.733005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:22.732958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:22.879829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:22.879754 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tpjp6" event={"ID":"a79cdc8f-6b2d-4623-b321-be0088cfad25","Type":"ContainerStarted","Data":"8da5e981344ef8deb3171c2ab63d31bc544381f845c8a376faf478f6e0dcf693"} Nov 01 21:45:23.101458 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:23.101421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:23.234011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:23.233935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:45:48.856191 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:45:48.856153 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:45:55.738331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:45:55.738291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:07.074519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:07.074481 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:07.197522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:07.197491 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:08.075479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:08.075425 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:08.207006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:08.206968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:09.638393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:09.638361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:09.770268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:09.770236 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:18.855841 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:46:18.855760 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:46:19.756550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:19.756517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:19.887127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:19.887086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:26.208312 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:26.208271 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:26.330160 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:26.330122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:37.083931 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:46:37.083904 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b9dd27_1e90_4edc_8646_3e7979ef591a.slice/crio-conmon-f05f7255503027926fc18d4abbef5cbfce3cf8e61f52379d80ffa1d2ebe01f72.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:46:37.167544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:37.167516 5036 generic.go:356] "Generic (PLEG): container finished" podID="f7b9dd27-1e90-4edc-8646-3e7979ef591a" containerID="f05f7255503027926fc18d4abbef5cbfce3cf8e61f52379d80ffa1d2ebe01f72" exitCode=0 Nov 01 21:46:37.167645 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:37.167561 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" event={"ID":"f7b9dd27-1e90-4edc-8646-3e7979ef591a","Type":"ContainerDied","Data":"f05f7255503027926fc18d4abbef5cbfce3cf8e61f52379d80ffa1d2ebe01f72"} Nov 01 21:46:38.298479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.298455 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:46:38.377804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.377775 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\") pod \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " Nov 01 21:46:38.377932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.377809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\") pod \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " Nov 01 21:46:38.377932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.377855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\") pod \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " Nov 01 21:46:38.377932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.377883 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\" (UniqueName: \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\") pod \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\" (UID: \"f7b9dd27-1e90-4edc-8646-3e7979ef591a\") " Nov 01 21:46:38.378208 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.378181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path" pod "f7b9dd27-1e90-4edc-8646-3e7979ef591a" (UID: "f7b9dd27-1e90-4edc-8646-3e7979ef591a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:46:38.378244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.378191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle" pod "f7b9dd27-1e90-4edc-8646-3e7979ef591a" (UID: "f7b9dd27-1e90-4edc-8646-3e7979ef591a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:38.378277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.378251 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle" pod "f7b9dd27-1e90-4edc-8646-3e7979ef591a" (UID: "f7b9dd27-1e90-4edc-8646-3e7979ef591a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:38.379869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.379851 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq" pod "f7b9dd27-1e90-4edc-8646-3e7979ef591a" (UID: "f7b9dd27-1e90-4edc-8646-3e7979ef591a"). InnerVolumeSpecName "kube-api-access-7zpgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:46:38.478704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.478677 5036 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f7b9dd27-1e90-4edc-8646-3e7979ef591a-archives-path\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:38.478704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.478700 5036 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-service-ca-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:38.478898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.478717 5036 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b9dd27-1e90-4edc-8646-3e7979ef591a-trusted-ca-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:38.478898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:38.478730 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpgq\" (UniqueName: \"kubernetes.io/projected/f7b9dd27-1e90-4edc-8646-3e7979ef591a-kube-api-access-7zpgq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:39.176196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:39.176167 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" Nov 01 21:46:39.176354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:39.176184 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vhpgm-lfdc2" event={"ID":"f7b9dd27-1e90-4edc-8646-3e7979ef591a","Type":"ContainerDied","Data":"b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4"} Nov 01 21:46:39.176354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:39.176254 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d41f789679a13704f26ea836a0915145740ea73cb6f023cf2ee03a6f5f4df4" Nov 01 21:46:39.679822 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:39.679787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:39.802125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:39.802088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:42.924436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:42.924399 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:48.856256 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:46:48.856218 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:46:55.794992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:46:55.794944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:12.435929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:12.435889 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:12.562974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:12.562941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:16.428337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:16.428306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:18.855979 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:47:18.855938 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:47:48.856776 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:47:48.856697 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:47:55.849131 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:55.849092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:59.625409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:59.625373 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:59.766707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:47:59.766671 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:01.109469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:01.109378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:01.251816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:01.251784 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:02.497855 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:02.497819 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:03.771834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.771802 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db"] Nov 01 21:48:03.772202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772099 5036 cpu_manager.go:414] "containerMap: removing container" podUID="793bff98-0107-4731-a6a0-5747f9303443" containerName="insights-gathering" Nov 01 21:48:03.772202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772110 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793bff98-0107-4731-a6a0-5747f9303443" containerName="insights-gathering" Nov 01 21:48:03.772202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772129 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f7b9dd27-1e90-4edc-8646-3e7979ef591a" containerName="insights-gathering" Nov 01 21:48:03.772202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772134 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b9dd27-1e90-4edc-8646-3e7979ef591a" containerName="insights-gathering" Nov 01 21:48:03.772202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772200 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b9dd27-1e90-4edc-8646-3e7979ef591a" containerName="insights-gathering" Nov 01 21:48:03.772360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.772211 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="793bff98-0107-4731-a6a0-5747f9303443" containerName="insights-gathering" Nov 01 21:48:03.775388 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.775370 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.780761 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.780726 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 21:48:03.780761 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.780727 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wzwh2\"" Nov 01 21:48:03.780902 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.780729 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 21:48:03.784658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.784610 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db"] Nov 01 21:48:03.870424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.870397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxtc\" (UniqueName: \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.870551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.870432 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.870551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.870490 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.971122 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.971095 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxtc\" (UniqueName: \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.971218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.971128 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.971218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.971151 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.971460 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.971428 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.971556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.971537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:03.984095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:03.984072 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxtc\" (UniqueName: \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:04.091796 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:04.091744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:04.364957 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:04.364884 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db"] Nov 01 21:48:04.367914 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:04.367882 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f59283_28a9_47ea_8f1b_514380bfce48.slice/crio-98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f WatchSource:0}: Error finding container 98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f: Status 404 returned error can't find the container with id 98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f Nov 01 21:48:04.495496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:04.495467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" event={"ID":"a8f59283-28a9-47ea-8f1b-514380bfce48","Type":"ContainerStarted","Data":"98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f"} Nov 01 21:48:05.443525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.443486 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf"] Nov 01 21:48:05.447145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.447120 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.454882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.454856 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7lvv8\"" Nov 01 21:48:05.456813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.456784 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf"] Nov 01 21:48:05.482435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.482407 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.482584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.482472 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlt2\" (UniqueName: \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.482649 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.482597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.540196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.540165 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf"] Nov 01 21:48:05.543795 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.543775 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.547765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.547745 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-nwtmn\"" Nov 01 21:48:05.553486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.553437 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf"] Nov 01 21:48:05.583185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.583299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583246 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.583361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583307 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8zp\" (UniqueName: \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.583408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.583476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583416 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.583476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583460 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlt2\" (UniqueName: \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.583579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583540 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.583748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.583728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.596358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.596330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlt2\" (UniqueName: \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.644404 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.644379 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg"] Nov 01 21:48:05.648465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.648422 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.656721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.656518 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lvcjn\"" Nov 01 21:48:05.658686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.658657 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg"] Nov 01 21:48:05.683812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.683790 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.683944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.683827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.683944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.683878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8zp\" (UniqueName: \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.683944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.683929 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.684058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.684013 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.684095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.684084 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f8p\" (UniqueName: \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.684167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.684148 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.684364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.684350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.697484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.697411 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8zp\" (UniqueName: \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.743760 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.743734 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg"] Nov 01 21:48:05.747670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.747649 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.755968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.755943 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-d566l\"" Nov 01 21:48:05.757980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.757386 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg"] Nov 01 21:48:05.766109 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.766090 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:05.785115 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785078 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.785212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785133 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f8p\" (UniqueName: \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.785274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785213 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.785274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785267 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9ts\" (UniqueName: \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.785382 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785303 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.785436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785415 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.785703 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.785750 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.785737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.797500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.797480 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f8p\" (UniqueName: \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.862584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.862559 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:05.886292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.886257 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9ts\" (UniqueName: \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.886413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.886310 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.886413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.886392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.886801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.886778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.886801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.886795 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.900051 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.900032 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9ts\" (UniqueName: \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:05.968850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.968791 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:05.985222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:05.985197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:06.048030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.047999 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf"] Nov 01 21:48:06.049716 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:06.049687 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd3b2ae_056a_40ed_b55a_8352310fe99e.slice/crio-b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b WatchSource:0}: Error finding container b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b: Status 404 returned error can't find the container with id b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b Nov 01 21:48:06.066654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.066630 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:06.114412 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.114378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:06.149361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.149336 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf"] Nov 01 21:48:06.152218 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:06.152189 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb960255e_ca94_4ffd_9e80_50b51fe90462.slice/crio-7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591 WatchSource:0}: Error finding container 7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591: Status 404 returned error can't find the container with id 7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591 Nov 01 21:48:06.249467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.249291 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg"] Nov 01 21:48:06.251633 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:06.251602 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aed172e_34e3_46ed_8838_618da9b7f5f2.slice/crio-6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7 WatchSource:0}: Error finding container 6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7: Status 404 returned error can't find the container with id 6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7 Nov 01 21:48:06.341958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.341925 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg"] Nov 01 21:48:06.345045 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:06.345011 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd537923f_8833_466c_b23a_abd2ad4efa9f.slice/crio-ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c WatchSource:0}: Error finding container ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c: Status 404 returned error can't find the container with id ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c Nov 01 21:48:06.505071 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.504989 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" event={"ID":"b960255e-ca94-4ffd-9e80-50b51fe90462","Type":"ContainerStarted","Data":"7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591"} Nov 01 21:48:06.506350 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.506321 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" event={"ID":"d537923f-8833-466c-b23a-abd2ad4efa9f","Type":"ContainerStarted","Data":"ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c"} Nov 01 21:48:06.507683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.507614 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" event={"ID":"3aed172e-34e3-46ed-8838-618da9b7f5f2","Type":"ContainerStarted","Data":"6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7"} Nov 01 21:48:06.508706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.508681 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerStarted","Data":"b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b"} Nov 01 21:48:06.846040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.844128 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j"] Nov 01 21:48:06.848518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.848126 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.853378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.852720 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qkp5d\"" Nov 01 21:48:06.861440 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.861411 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j"] Nov 01 21:48:06.894068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.894042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.894218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.894093 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.894218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.894166 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn79\" (UniqueName: \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.991551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.991513 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 21:48:06.995368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.995287 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.995368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.995336 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.995632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.995395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn79\" (UniqueName: \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.996065 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.996039 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.996287 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.996269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:06.996515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:06.996500 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.004469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.003538 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 21:48:07.004469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.003647 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tfhbq\"" Nov 01 21:48:07.018015 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.017989 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 21:48:07.018976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.018952 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn79\" (UniqueName: \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:07.076222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.074611 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw"] Nov 01 21:48:07.078970 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.078484 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.087473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.084099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:07.095376 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.093611 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw"] Nov 01 21:48:07.095887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.095844 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.095982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.095883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98kx\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-kube-api-access-f98kx\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.095982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.095917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.095982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.095973 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096000 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096027 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096060 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096177 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096250 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096272 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8zd\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.100647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.096404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.107029 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.106740 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-w9t5r\"" Nov 01 21:48:07.171050 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.171016 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:07.196831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196798 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.196948 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196856 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.196948 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196886 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.196948 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196914 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.196948 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196944 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.196984 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.197154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197062 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.197154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197121 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197163 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8zd\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197230 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197272 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197302 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98kx\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-kube-api-access-f98kx\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197383 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.197551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.197412 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.198224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.198199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.198654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.198625 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.198993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.198962 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.199678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.199656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.199774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.199744 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.200012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.199991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.200712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.200405 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.200712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.200480 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.200712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.200668 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.200958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.200916 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.218142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.218087 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.222048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.222020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.222692 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.222647 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.225186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.224958 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a770aa91-03cc-4886-a3a9-3fb96262c2fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.225186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.225143 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.227347 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.227322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.228930 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.228904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98kx\" (UniqueName: \"kubernetes.io/projected/a770aa91-03cc-4886-a3a9-3fb96262c2fe-kube-api-access-f98kx\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw\" (UID: \"a770aa91-03cc-4886-a3a9-3fb96262c2fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.230508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.230462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8zd\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-p4lpt\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.242306 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.242283 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:07.316716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.316685 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:07.406635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.403633 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:07.494354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.494185 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j"] Nov 01 21:48:07.516898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.516868 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerStarted","Data":"8616132d9370e399e423a7316470eebfb8bb155326128dd02284ac73f4188667"} Nov 01 21:48:07.636343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.636240 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 21:48:07.649313 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:07.649283 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83870513_6b8e_4205_9823_87b037b374b8.slice/crio-84da9438db88f202442962372e43644700c1582eb116e779c68313fe3054b5c1 WatchSource:0}: Error finding container 84da9438db88f202442962372e43644700c1582eb116e779c68313fe3054b5c1: Status 404 returned error can't find the container with id 84da9438db88f202442962372e43644700c1582eb116e779c68313fe3054b5c1 Nov 01 21:48:07.704897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:07.704869 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw"] Nov 01 21:48:07.710086 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:07.710058 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda770aa91_03cc_4886_a3a9_3fb96262c2fe.slice/crio-6123661bd749ca94b6bb47d1c54214b81da1ef4d97b11338f6012ce1acba46cf WatchSource:0}: Error finding container 6123661bd749ca94b6bb47d1c54214b81da1ef4d97b11338f6012ce1acba46cf: Status 404 returned error can't find the container with id 6123661bd749ca94b6bb47d1c54214b81da1ef4d97b11338f6012ce1acba46cf Nov 01 21:48:08.532290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:08.532245 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" event={"ID":"a770aa91-03cc-4886-a3a9-3fb96262c2fe","Type":"ContainerStarted","Data":"6123661bd749ca94b6bb47d1c54214b81da1ef4d97b11338f6012ce1acba46cf"} Nov 01 21:48:08.537468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:08.537291 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerStarted","Data":"84da9438db88f202442962372e43644700c1582eb116e779c68313fe3054b5c1"} Nov 01 21:48:09.522102 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:09.522070 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:15.681674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.681088 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:15.681674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.681155 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:15.681674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.681184 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:15.853176 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.853128 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:15.853360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.853221 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:15.853360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:15.853262 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:48:16.599690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:16.598688 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" event={"ID":"a770aa91-03cc-4886-a3a9-3fb96262c2fe","Type":"ContainerStarted","Data":"1157598072f743acaa3e68b7e6c5d2e49afc6f21899415fb558b9f792e8f426d"} Nov 01 21:48:16.602802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:16.602144 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerStarted","Data":"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8"} Nov 01 21:48:16.627721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:16.627666 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" podStartSLOduration=1.48866301 podStartE2EDuration="9.627646397s" totalImagesPullingTime="8.138983387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:16.62482294 +0000 UTC m=+250.317803427" watchObservedRunningTime="2026-11-01 21:48:16.627646397 +0000 UTC m=+250.320626884" Nov 01 21:48:17.317596 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.317571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:17.323628 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.323564 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:17.347521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.347436 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podStartSLOduration=3.324331689 podStartE2EDuration="11.347419082s" totalImagesPullingTime="8.023087393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:16.649502604 +0000 UTC m=+250.342483094" watchObservedRunningTime="2026-11-01 21:48:17.347419082 +0000 UTC m=+251.040399565" Nov 01 21:48:17.404861 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.404807 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:17.410800 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.410763 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:17.613481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.613112 5036 generic.go:356] "Generic (PLEG): container finished" podID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerID="fe2cb97ec6b8126e7ae86c18b6e5b9bb44fb802ddbd05e98a3f150a8d197ba49" exitCode=0 Nov 01 21:48:17.613481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.613326 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" event={"ID":"a8f59283-28a9-47ea-8f1b-514380bfce48","Type":"ContainerDied","Data":"fe2cb97ec6b8126e7ae86c18b6e5b9bb44fb802ddbd05e98a3f150a8d197ba49"} Nov 01 21:48:17.621335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.620331 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:17.621335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.621169 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:17.622298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.622032 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw" Nov 01 21:48:17.622298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:17.622232 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:48:18.868391 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:48:18.868342 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:48:19.627677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:19.627608 5036 generic.go:356] "Generic (PLEG): container finished" podID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerID="513928eb28703ec25e858d142cb671625ce3654dea8f8865b91861d69cec9e2d" exitCode=0 Nov 01 21:48:19.627929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:19.627898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerDied","Data":"513928eb28703ec25e858d142cb671625ce3654dea8f8865b91861d69cec9e2d"} Nov 01 21:48:19.631476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:19.630993 5036 generic.go:356] "Generic (PLEG): container finished" podID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerID="122a049b00c6c69ade7ffaad33701fba461b16ff33f02ee4927f39dfa74b6bce" exitCode=0 Nov 01 21:48:19.631476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:19.631084 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" event={"ID":"a8f59283-28a9-47ea-8f1b-514380bfce48","Type":"ContainerDied","Data":"122a049b00c6c69ade7ffaad33701fba461b16ff33f02ee4927f39dfa74b6bce"} Nov 01 21:48:20.638410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.638376 5036 generic.go:356] "Generic (PLEG): container finished" podID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerID="c6cb30b4a910e988b9289cd52851dba5efd79e576460d9d6dff0f1f853fea65f" exitCode=0 Nov 01 21:48:20.638934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.638506 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" event={"ID":"b960255e-ca94-4ffd-9e80-50b51fe90462","Type":"ContainerDied","Data":"c6cb30b4a910e988b9289cd52851dba5efd79e576460d9d6dff0f1f853fea65f"} Nov 01 21:48:20.640313 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.640285 5036 generic.go:356] "Generic (PLEG): container finished" podID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerID="f605a10fdb3f049d61e30d9306231b40d5353a4e4d62e6688d00f1332da7b68c" exitCode=0 Nov 01 21:48:20.640424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.640344 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" event={"ID":"d537923f-8833-466c-b23a-abd2ad4efa9f","Type":"ContainerDied","Data":"f605a10fdb3f049d61e30d9306231b40d5353a4e4d62e6688d00f1332da7b68c"} Nov 01 21:48:20.642676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.642541 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerID="851d75ac064527e79c8c6e67cff8f456f93e48df6b37ce261d266e2d1a2c922b" exitCode=0 Nov 01 21:48:20.642676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:20.642632 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" event={"ID":"3aed172e-34e3-46ed-8838-618da9b7f5f2","Type":"ContainerDied","Data":"851d75ac064527e79c8c6e67cff8f456f93e48df6b37ce261d266e2d1a2c922b"} Nov 01 21:48:21.656608 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:21.656567 5036 generic.go:356] "Generic (PLEG): container finished" podID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerID="d46d7afca93fb3f997c279c74878ec4eb68e4d27dfe99fbb88dcbfe20535bcf3" exitCode=0 Nov 01 21:48:21.657080 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:21.656718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerDied","Data":"d46d7afca93fb3f997c279c74878ec4eb68e4d27dfe99fbb88dcbfe20535bcf3"} Nov 01 21:48:21.658976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:21.658945 5036 generic.go:356] "Generic (PLEG): container finished" podID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerID="c45e9aaab230c8b7978f4542ce26849c02b1be619b2d583eb65f1fe5e5be3734" exitCode=0 Nov 01 21:48:21.659124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:21.658998 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerDied","Data":"c45e9aaab230c8b7978f4542ce26849c02b1be619b2d583eb65f1fe5e5be3734"} Nov 01 21:48:22.667559 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.667520 5036 generic.go:356] "Generic (PLEG): container finished" podID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerID="37dc08714c3ea8cedb6134a48fff831ad1d4009de353590e33dcb6516ab58553" exitCode=0 Nov 01 21:48:22.667979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.667600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" event={"ID":"b960255e-ca94-4ffd-9e80-50b51fe90462","Type":"ContainerDied","Data":"37dc08714c3ea8cedb6134a48fff831ad1d4009de353590e33dcb6516ab58553"} Nov 01 21:48:22.670930 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.670882 5036 generic.go:356] "Generic (PLEG): container finished" podID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerID="2123c1194f009412b71f0a470e3d4bcce304286549b67c2038a6986a470ef91e" exitCode=0 Nov 01 21:48:22.671041 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.670925 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" event={"ID":"d537923f-8833-466c-b23a-abd2ad4efa9f","Type":"ContainerDied","Data":"2123c1194f009412b71f0a470e3d4bcce304286549b67c2038a6986a470ef91e"} Nov 01 21:48:22.673601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.673502 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerID="d5c1a3e9e7e0cfc28bd3ccd04021dbaf09bedcc817a4e8c6dbfdc8a2036e5764" exitCode=0 Nov 01 21:48:22.673601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:22.673559 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" event={"ID":"3aed172e-34e3-46ed-8838-618da9b7f5f2","Type":"ContainerDied","Data":"d5c1a3e9e7e0cfc28bd3ccd04021dbaf09bedcc817a4e8c6dbfdc8a2036e5764"} Nov 01 21:48:25.698291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:25.698259 5036 generic.go:356] "Generic (PLEG): container finished" podID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerID="3a43af18b12861f93d738883720df458ebb56de6e77881bbe86887be73bbf250" exitCode=0 Nov 01 21:48:25.698847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:25.698397 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerDied","Data":"3a43af18b12861f93d738883720df458ebb56de6e77881bbe86887be73bbf250"} Nov 01 21:48:27.867664 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:27.867560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:29.836125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:29.836082 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:32.756551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:32.756099 5036 generic.go:356] "Generic (PLEG): container finished" podID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerID="35a1413370b4722a5e6361f3c2697ed8b8efe8863a21fdf1230ff1ead4293f6b" exitCode=0 Nov 01 21:48:32.757045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:32.756592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" event={"ID":"a8f59283-28a9-47ea-8f1b-514380bfce48","Type":"ContainerDied","Data":"35a1413370b4722a5e6361f3c2697ed8b8efe8863a21fdf1230ff1ead4293f6b"} Nov 01 21:48:33.766100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:33.766063 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerStarted","Data":"abe3807ebefb07ca575b95bb5ad05a16736519a8d24415c60c30a328d6a206da"} Nov 01 21:48:33.770860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:33.770831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerStarted","Data":"b6711832dfa504670851bbb14bc103621fb643194b4fc2b4e2b0734233a35276"} Nov 01 21:48:33.811335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:33.811236 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" podStartSLOduration=2.197637675 podStartE2EDuration="28.811222496s" totalImagesPullingTime="26.613584821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:33.790163634 +0000 UTC m=+267.483144121" watchObservedRunningTime="2026-11-01 21:48:33.811222496 +0000 UTC m=+267.504202976" Nov 01 21:48:33.811521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:33.811474 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" podStartSLOduration=3.4535505349999998 podStartE2EDuration="27.811467686s" totalImagesPullingTime="24.357917151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:33.810403592 +0000 UTC m=+267.503384079" watchObservedRunningTime="2026-11-01 21:48:33.811467686 +0000 UTC m=+267.504448167" Nov 01 21:48:33.929925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:33.929611 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:34.081653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.081567 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\") pod \"a8f59283-28a9-47ea-8f1b-514380bfce48\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " Nov 01 21:48:34.081653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.081647 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\" (UniqueName: \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\") pod \"a8f59283-28a9-47ea-8f1b-514380bfce48\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " Nov 01 21:48:34.081890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.081722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\") pod \"a8f59283-28a9-47ea-8f1b-514380bfce48\" (UID: \"a8f59283-28a9-47ea-8f1b-514380bfce48\") " Nov 01 21:48:34.082421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.082390 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle" pod "a8f59283-28a9-47ea-8f1b-514380bfce48" (UID: "a8f59283-28a9-47ea-8f1b-514380bfce48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:34.086476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.086438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util" pod "a8f59283-28a9-47ea-8f1b-514380bfce48" (UID: "a8f59283-28a9-47ea-8f1b-514380bfce48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:34.087666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.087640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc" pod "a8f59283-28a9-47ea-8f1b-514380bfce48" (UID: "a8f59283-28a9-47ea-8f1b-514380bfce48"). InnerVolumeSpecName "kube-api-access-mkxtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:34.182949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.182913 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:34.182949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.182949 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8f59283-28a9-47ea-8f1b-514380bfce48-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:34.183151 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.182961 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxtc\" (UniqueName: \"kubernetes.io/projected/a8f59283-28a9-47ea-8f1b-514380bfce48-kube-api-access-mkxtc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:34.778339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.778314 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" Nov 01 21:48:34.778720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.778414 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719t5db" event={"ID":"a8f59283-28a9-47ea-8f1b-514380bfce48","Type":"ContainerDied","Data":"98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f"} Nov 01 21:48:34.778720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.778484 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98739850013af36c4eee10672d02bfa399f6ba2ce54644b5f428edd2a541399f" Nov 01 21:48:34.781736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.781709 5036 generic.go:356] "Generic (PLEG): container finished" podID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerID="b6711832dfa504670851bbb14bc103621fb643194b4fc2b4e2b0734233a35276" exitCode=0 Nov 01 21:48:34.781851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.781792 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerDied","Data":"b6711832dfa504670851bbb14bc103621fb643194b4fc2b4e2b0734233a35276"} Nov 01 21:48:34.784429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.784407 5036 generic.go:356] "Generic (PLEG): container finished" podID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerID="abe3807ebefb07ca575b95bb5ad05a16736519a8d24415c60c30a328d6a206da" exitCode=0 Nov 01 21:48:34.784545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:34.784483 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerDied","Data":"abe3807ebefb07ca575b95bb5ad05a16736519a8d24415c60c30a328d6a206da"} Nov 01 21:48:35.791175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.791142 5036 generic.go:356] "Generic (PLEG): container finished" podID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerID="b9e068188d491cb432f3707fbfb3cb6697eab882b2fb9b31322d270515d47ddb" exitCode=0 Nov 01 21:48:35.791572 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.791227 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" event={"ID":"b960255e-ca94-4ffd-9e80-50b51fe90462","Type":"ContainerDied","Data":"b9e068188d491cb432f3707fbfb3cb6697eab882b2fb9b31322d270515d47ddb"} Nov 01 21:48:35.793310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.793284 5036 generic.go:356] "Generic (PLEG): container finished" podID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerID="ce142b53d5ff8833cfcde3e140f11f9833cb824fb0028632a083bb55aadb6b89" exitCode=0 Nov 01 21:48:35.793407 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.793369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" event={"ID":"d537923f-8833-466c-b23a-abd2ad4efa9f","Type":"ContainerDied","Data":"ce142b53d5ff8833cfcde3e140f11f9833cb824fb0028632a083bb55aadb6b89"} Nov 01 21:48:35.795453 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.795419 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerID="4b5b831280c842fdfcaa685052eba6f7aee0903b9045c63dbfd8e31cdc8e5dba" exitCode=0 Nov 01 21:48:35.795553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.795481 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" event={"ID":"3aed172e-34e3-46ed-8838-618da9b7f5f2","Type":"ContainerDied","Data":"4b5b831280c842fdfcaa685052eba6f7aee0903b9045c63dbfd8e31cdc8e5dba"} Nov 01 21:48:35.941840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.941820 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:35.944464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:35.944426 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:36.099603 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099528 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\" (UniqueName: \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\") pod \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " Nov 01 21:48:36.099747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099607 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\") pod \"b53458e3-53c1-4911-9dc0-b4477c9f016b\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " Nov 01 21:48:36.099747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\") pod \"b53458e3-53c1-4911-9dc0-b4477c9f016b\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " Nov 01 21:48:36.099747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\" (UniqueName: \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\") pod \"b53458e3-53c1-4911-9dc0-b4477c9f016b\" (UID: \"b53458e3-53c1-4911-9dc0-b4477c9f016b\") " Nov 01 21:48:36.099870 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099773 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\") pod \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " Nov 01 21:48:36.099870 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.099838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\") pod \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\" (UID: \"5fd3b2ae-056a-40ed-b55a-8352310fe99e\") " Nov 01 21:48:36.100470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.100429 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle" pod "b53458e3-53c1-4911-9dc0-b4477c9f016b" (UID: "b53458e3-53c1-4911-9dc0-b4477c9f016b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:36.100674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.100644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle" pod "5fd3b2ae-056a-40ed-b55a-8352310fe99e" (UID: "5fd3b2ae-056a-40ed-b55a-8352310fe99e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:36.101900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.101870 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79" pod "b53458e3-53c1-4911-9dc0-b4477c9f016b" (UID: "b53458e3-53c1-4911-9dc0-b4477c9f016b"). InnerVolumeSpecName "kube-api-access-ljn79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:36.102023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.102002 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2" pod "5fd3b2ae-056a-40ed-b55a-8352310fe99e" (UID: "5fd3b2ae-056a-40ed-b55a-8352310fe99e"). InnerVolumeSpecName "kube-api-access-7hlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:36.106346 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.106322 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util" pod "5fd3b2ae-056a-40ed-b55a-8352310fe99e" (UID: "5fd3b2ae-056a-40ed-b55a-8352310fe99e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:36.106423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.106400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util" pod "b53458e3-53c1-4911-9dc0-b4477c9f016b" (UID: "b53458e3-53c1-4911-9dc0-b4477c9f016b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:36.201155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201131 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.201155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201154 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlt2\" (UniqueName: \"kubernetes.io/projected/5fd3b2ae-056a-40ed-b55a-8352310fe99e-kube-api-access-7hlt2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.201273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201164 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.201273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201173 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b53458e3-53c1-4911-9dc0-b4477c9f016b-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.201273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201181 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn79\" (UniqueName: \"kubernetes.io/projected/b53458e3-53c1-4911-9dc0-b4477c9f016b-kube-api-access-ljn79\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.201273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.201190 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd3b2ae-056a-40ed-b55a-8352310fe99e-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:36.801225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.801198 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" Nov 01 21:48:36.801639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.801214 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azz9zf" event={"ID":"5fd3b2ae-056a-40ed-b55a-8352310fe99e","Type":"ContainerDied","Data":"b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b"} Nov 01 21:48:36.801639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.801271 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d501c4b0c539756589bae916f9b979383c47737387af430650ad6bc288409b" Nov 01 21:48:36.803286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.803265 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" Nov 01 21:48:36.803410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.803308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxs8j" event={"ID":"b53458e3-53c1-4911-9dc0-b4477c9f016b","Type":"ContainerDied","Data":"8616132d9370e399e423a7316470eebfb8bb155326128dd02284ac73f4188667"} Nov 01 21:48:36.803410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.803336 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8616132d9370e399e423a7316470eebfb8bb155326128dd02284ac73f4188667" Nov 01 21:48:36.957487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.957461 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:36.977539 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.977516 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:36.984436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:36.984414 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:37.108154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108081 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\") pod \"b960255e-ca94-4ffd-9e80-50b51fe90462\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " Nov 01 21:48:37.108154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\") pod \"3aed172e-34e3-46ed-8838-618da9b7f5f2\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " Nov 01 21:48:37.108154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\") pod \"b960255e-ca94-4ffd-9e80-50b51fe90462\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " Nov 01 21:48:37.108371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108196 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\") pod \"3aed172e-34e3-46ed-8838-618da9b7f5f2\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " Nov 01 21:48:37.108371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\" (UniqueName: \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\") pod \"3aed172e-34e3-46ed-8838-618da9b7f5f2\" (UID: \"3aed172e-34e3-46ed-8838-618da9b7f5f2\") " Nov 01 21:48:37.108371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\") pod \"d537923f-8833-466c-b23a-abd2ad4efa9f\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " Nov 01 21:48:37.108371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108294 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\" (UniqueName: \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\") pod \"d537923f-8833-466c-b23a-abd2ad4efa9f\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " Nov 01 21:48:37.108371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108313 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\" (UniqueName: \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\") pod \"b960255e-ca94-4ffd-9e80-50b51fe90462\" (UID: \"b960255e-ca94-4ffd-9e80-50b51fe90462\") " Nov 01 21:48:37.108822 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108802 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\") pod \"d537923f-8833-466c-b23a-abd2ad4efa9f\" (UID: \"d537923f-8833-466c-b23a-abd2ad4efa9f\") " Nov 01 21:48:37.108903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108817 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle" pod "3aed172e-34e3-46ed-8838-618da9b7f5f2" (UID: "3aed172e-34e3-46ed-8838-618da9b7f5f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.108974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.108912 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle" pod "b960255e-ca94-4ffd-9e80-50b51fe90462" (UID: "b960255e-ca94-4ffd-9e80-50b51fe90462"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.109319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.109295 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.109408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.109323 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.109600 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.109563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle" pod "d537923f-8833-466c-b23a-abd2ad4efa9f" (UID: "d537923f-8833-466c-b23a-abd2ad4efa9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.110856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.110830 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p" pod "3aed172e-34e3-46ed-8838-618da9b7f5f2" (UID: "3aed172e-34e3-46ed-8838-618da9b7f5f2"). InnerVolumeSpecName "kube-api-access-s8f8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:37.110946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.110899 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp" pod "b960255e-ca94-4ffd-9e80-50b51fe90462" (UID: "b960255e-ca94-4ffd-9e80-50b51fe90462"). InnerVolumeSpecName "kube-api-access-4d8zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:37.111153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.111134 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts" pod "d537923f-8833-466c-b23a-abd2ad4efa9f" (UID: "d537923f-8833-466c-b23a-abd2ad4efa9f"). InnerVolumeSpecName "kube-api-access-ct9ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:37.113234 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.113212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util" pod "b960255e-ca94-4ffd-9e80-50b51fe90462" (UID: "b960255e-ca94-4ffd-9e80-50b51fe90462"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.114968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.114948 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util" pod "3aed172e-34e3-46ed-8838-618da9b7f5f2" (UID: "3aed172e-34e3-46ed-8838-618da9b7f5f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.115439 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.115419 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util" pod "d537923f-8833-466c-b23a-abd2ad4efa9f" (UID: "d537923f-8833-466c-b23a-abd2ad4efa9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:37.210647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210623 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8zp\" (UniqueName: \"kubernetes.io/projected/b960255e-ca94-4ffd-9e80-50b51fe90462-kube-api-access-4d8zp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210647 5036 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-bundle\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210658 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b960255e-ca94-4ffd-9e80-50b51fe90462-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210666 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aed172e-34e3-46ed-8838-618da9b7f5f2-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210674 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f8p\" (UniqueName: \"kubernetes.io/projected/3aed172e-34e3-46ed-8838-618da9b7f5f2-kube-api-access-s8f8p\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210681 5036 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d537923f-8833-466c-b23a-abd2ad4efa9f-util\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.210763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.210690 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9ts\" (UniqueName: \"kubernetes.io/projected/d537923f-8833-466c-b23a-abd2ad4efa9f-kube-api-access-ct9ts\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:37.815083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.815047 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" event={"ID":"b960255e-ca94-4ffd-9e80-50b51fe90462","Type":"ContainerDied","Data":"7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591"} Nov 01 21:48:37.815510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.815128 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d3e10f38279219f76a784e9bd5fb014a4818354fc6744b76640d3515b411591" Nov 01 21:48:37.815510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.815061 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswqbf" Nov 01 21:48:37.817018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.816993 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" event={"ID":"d537923f-8833-466c-b23a-abd2ad4efa9f","Type":"ContainerDied","Data":"ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c"} Nov 01 21:48:37.817139 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.817025 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7swxxg" Nov 01 21:48:37.817139 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.817034 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1fc6ebf8d324430ed9f52be712e5aac66e1a3853249990166a29cd22ab5c7c" Nov 01 21:48:37.818908 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.818891 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" Nov 01 21:48:37.818993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.818902 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69rdkg" event={"ID":"3aed172e-34e3-46ed-8838-618da9b7f5f2","Type":"ContainerDied","Data":"6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7"} Nov 01 21:48:37.818993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:37.818934 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eab83889c073953c3696464a40af4432b0a6e4552082a32c7d000e708daf3e7" Nov 01 21:48:38.799419 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:38.799387 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:38.927646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:38.927610 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:39.644831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.644797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:39.728211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728171 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk"] Nov 01 21:48:39.728599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728581 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="pull" Nov 01 21:48:39.728599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728599 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="pull" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728616 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728623 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728636 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="util" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728644 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="util" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728653 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728661 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728672 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="pull" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728681 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="pull" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728694 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="util" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728700 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="util" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728711 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="pull" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728717 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="pull" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728727 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728734 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728747 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728753 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="extract" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728769 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="util" Nov 01 21:48:39.728774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728775 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728789 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728796 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728815 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728822 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728832 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728838 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728848 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728854 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728866 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728873 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728886 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728893 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="util" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728902 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728909 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728923 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.728929 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="pull" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729016 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d537923f-8833-466c-b23a-abd2ad4efa9f" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729030 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b960255e-ca94-4ffd-9e80-50b51fe90462" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729042 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53458e3-53c1-4911-9dc0-b4477c9f016b" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729050 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aed172e-34e3-46ed-8838-618da9b7f5f2" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729059 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd3b2ae-056a-40ed-b55a-8352310fe99e" containerName="extract" Nov 01 21:48:39.729669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.729068 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f59283-28a9-47ea-8f1b-514380bfce48" containerName="extract" Nov 01 21:48:39.760724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.760697 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk"] Nov 01 21:48:39.760724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.760716 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.867982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.867936 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 21:48:39.867982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.867943 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 21:48:39.867982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.867970 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 21:48:39.868154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.868018 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 21:48:39.868154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.868083 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wbcjm\"" Nov 01 21:48:39.868228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.868157 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 21:48:39.932103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932051 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.932103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.932541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932111 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.932541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932171 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvjn\" (UniqueName: \"kubernetes.io/projected/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-kube-api-access-qsvjn\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.932541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932325 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.932541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.932369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:39.958273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:39.958242 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:40.033256 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033228 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033261 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvjn\" (UniqueName: \"kubernetes.io/projected/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-kube-api-access-qsvjn\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033318 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033346 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.033683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.033662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.035863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.035842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.035863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.035852 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.036018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.035931 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.036148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.036128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.045903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.045882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvjn\" (UniqueName: \"kubernetes.io/projected/4b4875ba-473a-4ff3-a1ad-f3a6581c4090-kube-api-access-qsvjn\") pod \"opendatahub-operator-controller-manager-98c48f84c-86bbk\" (UID: \"4b4875ba-473a-4ff3-a1ad-f3a6581c4090\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.084336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.084297 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:40.110654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.110630 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:40.219956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.219894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:40.448224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.448190 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk"] Nov 01 21:48:40.451361 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:40.451333 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4875ba_473a_4ff3_a1ad_f3a6581c4090.slice/crio-a73d35e8253077c946cff206baf85bb7e0984e5435f3d0a3ccb8f7a7482f6481 WatchSource:0}: Error finding container a73d35e8253077c946cff206baf85bb7e0984e5435f3d0a3ccb8f7a7482f6481: Status 404 returned error can't find the container with id a73d35e8253077c946cff206baf85bb7e0984e5435f3d0a3ccb8f7a7482f6481 Nov 01 21:48:40.833525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:40.833489 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" event={"ID":"4b4875ba-473a-4ff3-a1ad-f3a6581c4090","Type":"ContainerStarted","Data":"a73d35e8253077c946cff206baf85bb7e0984e5435f3d0a3ccb8f7a7482f6481"} Nov 01 21:48:42.596486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:42.596433 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:42.822883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:42.822834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:43.768400 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:43.768366 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:43.902991 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:43.902894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:44.854568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:44.854488 5036 generic.go:356] "Generic (PLEG): container finished" podID="4b4875ba-473a-4ff3-a1ad-f3a6581c4090" containerID="e2d101707ce6d530093664cd21efc01d86e103448af73aa436e222ae2823177c" exitCode=0 Nov 01 21:48:44.854568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:44.854556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" event={"ID":"4b4875ba-473a-4ff3-a1ad-f3a6581c4090","Type":"ContainerDied","Data":"e2d101707ce6d530093664cd21efc01d86e103448af73aa436e222ae2823177c"} Nov 01 21:48:45.861048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:45.861016 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" event={"ID":"4b4875ba-473a-4ff3-a1ad-f3a6581c4090","Type":"ContainerStarted","Data":"daca25c8e1aecc8abf96a6770240d1fa8296c8fc057cdcdc50a8a67c9d510475"} Nov 01 21:48:45.861429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:45.861124 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:45.931186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:45.931143 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" podStartSLOduration=2.473818533 podStartE2EDuration="6.931129828s" totalImagesPullingTime="4.047408795s" totalInitContainerRuntime="409.9025ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:45.928187775 +0000 UTC m=+279.621168261" watchObservedRunningTime="2026-11-01 21:48:45.931129828 +0000 UTC m=+279.624110313" Nov 01 21:48:46.240840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.240770 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl"] Nov 01 21:48:46.244239 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.244220 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.255003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.254985 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jljds\"" Nov 01 21:48:46.271137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.271116 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl"] Nov 01 21:48:46.275181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.275161 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:48:46.277644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.277625 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 21:48:46.357890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.357858 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:46.387301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.387279 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b795608-da46-4c99-a308-4795190f88fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.387389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.387310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q79pq\" (UniqueName: \"kubernetes.io/projected/2b795608-da46-4c99-a308-4795190f88fa-kube-api-access-q79pq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.488693 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.488666 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b795608-da46-4c99-a308-4795190f88fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.488793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.488699 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q79pq\" (UniqueName: \"kubernetes.io/projected/2b795608-da46-4c99-a308-4795190f88fa-kube-api-access-q79pq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.489030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.489010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b795608-da46-4c99-a308-4795190f88fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.508139 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.508084 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q79pq\" (UniqueName: \"kubernetes.io/projected/2b795608-da46-4c99-a308-4795190f88fa-kube-api-access-q79pq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl\" (UID: \"2b795608-da46-4c99-a308-4795190f88fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.556118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.556098 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" Nov 01 21:48:46.841604 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.841580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl"] Nov 01 21:48:46.843577 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:46.843550 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b795608_da46_4c99_a308_4795190f88fa.slice/crio-820dc0fe2ab51b19550657c547f184482809aed70abcd369509946c8ba88e49b WatchSource:0}: Error finding container 820dc0fe2ab51b19550657c547f184482809aed70abcd369509946c8ba88e49b: Status 404 returned error can't find the container with id 820dc0fe2ab51b19550657c547f184482809aed70abcd369509946c8ba88e49b Nov 01 21:48:46.867124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:46.867095 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" event={"ID":"2b795608-da46-4c99-a308-4795190f88fa","Type":"ContainerStarted","Data":"820dc0fe2ab51b19550657c547f184482809aed70abcd369509946c8ba88e49b"} Nov 01 21:48:47.230355 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:47.230278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:47.361080 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:47.361042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:48.081783 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:48.081748 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:48.217153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:48.217118 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:48.858372 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:48:48.858340 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:48:49.589312 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:49.589276 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:49.881526 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:49.881422 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" event={"ID":"2b795608-da46-4c99-a308-4795190f88fa","Type":"ContainerStarted","Data":"0326828bb281bf080d42c58139d608dbd113f262ff44636ebd00c6dcc4628d0c"} Nov 01 21:48:49.910005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:49.909964 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ksdzl" podStartSLOduration=1.841086561 podStartE2EDuration="3.909953122s" totalImagesPullingTime="2.068866561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:49.909476209 +0000 UTC m=+283.602456709" watchObservedRunningTime="2026-11-01 21:48:49.909953122 +0000 UTC m=+283.602933608" Nov 01 21:48:50.742321 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:50.742281 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:50.878383 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:50.878349 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:52.619374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.619337 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz"] Nov 01 21:48:52.624655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.624637 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:52.633721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.633701 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6t887\"" Nov 01 21:48:52.633986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.633966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmzn\" (UniqueName: \"kubernetes.io/projected/60443ab7-2c7c-4246-aaa1-e6991846441f-kube-api-access-kdmzn\") pod \"dns-operator-controller-manager-8557cbdb84-vd5qz\" (UID: \"60443ab7-2c7c-4246-aaa1-e6991846441f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:52.638121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.638096 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz"] Nov 01 21:48:52.682968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.682943 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 21:48:52.682968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.682952 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 21:48:52.682968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.682963 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 21:48:52.734627 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.734604 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmzn\" (UniqueName: \"kubernetes.io/projected/60443ab7-2c7c-4246-aaa1-e6991846441f-kube-api-access-kdmzn\") pod \"dns-operator-controller-manager-8557cbdb84-vd5qz\" (UID: \"60443ab7-2c7c-4246-aaa1-e6991846441f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:52.768936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.768906 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmzn\" (UniqueName: \"kubernetes.io/projected/60443ab7-2c7c-4246-aaa1-e6991846441f-kube-api-access-kdmzn\") pod \"dns-operator-controller-manager-8557cbdb84-vd5qz\" (UID: \"60443ab7-2c7c-4246-aaa1-e6991846441f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:52.937009 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.936943 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:52.994760 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:52.994734 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:53.129828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:53.129791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:53.213483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:53.213437 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz"] Nov 01 21:48:53.215614 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:53.215575 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60443ab7_2c7c_4246_aaa1_e6991846441f.slice/crio-9e92ac7eece1f90318c26794af8bcb211616d08e7b03ce3354d44528c1d42277 WatchSource:0}: Error finding container 9e92ac7eece1f90318c26794af8bcb211616d08e7b03ce3354d44528c1d42277: Status 404 returned error can't find the container with id 9e92ac7eece1f90318c26794af8bcb211616d08e7b03ce3354d44528c1d42277 Nov 01 21:48:53.335697 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:53.335657 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:53.470973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:53.470882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:53.899703 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:53.899671 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" event={"ID":"60443ab7-2c7c-4246-aaa1-e6991846441f","Type":"ContainerStarted","Data":"9e92ac7eece1f90318c26794af8bcb211616d08e7b03ce3354d44528c1d42277"} Nov 01 21:48:54.217975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:54.217887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:54.513682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:54.513589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:55.925342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:55.925254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:56.431190 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.431149 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:56.869798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.869771 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-86bbk" Nov 01 21:48:56.918604 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.918573 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" event={"ID":"60443ab7-2c7c-4246-aaa1-e6991846441f","Type":"ContainerStarted","Data":"19b78718ea023c77b769c2d50bf55c7a528c8320d80df2999e03ca7a0163ab0f"} Nov 01 21:48:56.918774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.918693 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:48:56.922535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.922509 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvd26"] Nov 01 21:48:56.926116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.926093 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:56.939796 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.939775 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7ssvv\"" Nov 01 21:48:56.942405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.942384 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvd26"] Nov 01 21:48:56.945010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.944960 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" podStartSLOduration=1.917302051 podStartE2EDuration="4.944944796s" totalImagesPullingTime="3.027642745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:56.942938868 +0000 UTC m=+290.635919355" watchObservedRunningTime="2026-11-01 21:48:56.944944796 +0000 UTC m=+290.637925284" Nov 01 21:48:56.964073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:56.964019 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4cn\" (UniqueName: \"kubernetes.io/projected/dadece39-3d29-4a88-ace0-e0b9c0d7e2e7-kube-api-access-8k4cn\") pod \"authorino-operator-7b7bb4bc77-hvd26\" (UID: \"dadece39-3d29-4a88-ace0-e0b9c0d7e2e7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:57.065011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:57.064978 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4cn\" (UniqueName: \"kubernetes.io/projected/dadece39-3d29-4a88-ace0-e0b9c0d7e2e7-kube-api-access-8k4cn\") pod \"authorino-operator-7b7bb4bc77-hvd26\" (UID: \"dadece39-3d29-4a88-ace0-e0b9c0d7e2e7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:57.079259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:57.079226 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4cn\" (UniqueName: \"kubernetes.io/projected/dadece39-3d29-4a88-ace0-e0b9c0d7e2e7-kube-api-access-8k4cn\") pod \"authorino-operator-7b7bb4bc77-hvd26\" (UID: \"dadece39-3d29-4a88-ace0-e0b9c0d7e2e7\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:57.240118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:57.240043 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:57.518560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:57.518538 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvd26"] Nov 01 21:48:57.520927 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:48:57.520899 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadece39_3d29_4a88_ace0_e0b9c0d7e2e7.slice/crio-0eecd88d72c22d11d3704f322c2630ab5d0a528c86733f96d82d0a533c3d6856 WatchSource:0}: Error finding container 0eecd88d72c22d11d3704f322c2630ab5d0a528c86733f96d82d0a533c3d6856: Status 404 returned error can't find the container with id 0eecd88d72c22d11d3704f322c2630ab5d0a528c86733f96d82d0a533c3d6856 Nov 01 21:48:57.925632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:57.925594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" event={"ID":"dadece39-3d29-4a88-ace0-e0b9c0d7e2e7","Type":"ContainerStarted","Data":"0eecd88d72c22d11d3704f322c2630ab5d0a528c86733f96d82d0a533c3d6856"} Nov 01 21:48:58.610528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:58.610487 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:58.833819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:58.833783 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:59.454874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:59.454851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:59.669404 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:59.669321 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:59.936480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:59.936380 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" event={"ID":"dadece39-3d29-4a88-ace0-e0b9c0d7e2e7","Type":"ContainerStarted","Data":"acfccbb5fa1c9b00602eb8f6eafc490285eeae27bb559c3b73d74576f6a1c0c5"} Nov 01 21:48:59.936630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:59.936489 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:48:59.967155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:48:59.967115 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" podStartSLOduration=2.14623652 podStartE2EDuration="3.967102326s" totalImagesPullingTime="1.820865806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:59.965783219 +0000 UTC m=+293.658763705" watchObservedRunningTime="2026-11-01 21:48:59.967102326 +0000 UTC m=+293.660082806" Nov 01 21:49:00.761690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:00.761651 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:01.493005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.492969 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh"] Nov 01 21:49:01.496576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.496559 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:01.503832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.503811 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pmbnh\"" Nov 01 21:49:01.515270 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.515249 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh"] Nov 01 21:49:01.601732 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.601699 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl96\" (UniqueName: \"kubernetes.io/projected/fe02d9de-6509-43ff-a453-8f9373c24da8-kube-api-access-cpl96\") pod \"limitador-operator-controller-manager-5778bb4449-t29lh\" (UID: \"fe02d9de-6509-43ff-a453-8f9373c24da8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:01.702848 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.702818 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl96\" (UniqueName: \"kubernetes.io/projected/fe02d9de-6509-43ff-a453-8f9373c24da8-kube-api-access-cpl96\") pod \"limitador-operator-controller-manager-5778bb4449-t29lh\" (UID: \"fe02d9de-6509-43ff-a453-8f9373c24da8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:01.724365 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.724344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl96\" (UniqueName: \"kubernetes.io/projected/fe02d9de-6509-43ff-a453-8f9373c24da8-kube-api-access-cpl96\") pod \"limitador-operator-controller-manager-5778bb4449-t29lh\" (UID: \"fe02d9de-6509-43ff-a453-8f9373c24da8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:01.810124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.810099 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:01.908951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:01.908921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:02.035677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:02.035646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:02.082558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:02.082494 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh"] Nov 01 21:49:02.085756 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:02.085732 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe02d9de_6509_43ff_a453_8f9373c24da8.slice/crio-25606cf2916dc2a1c922aceed5eab7be07fcd0db5ca5b1c17a14883b050d3ce2 WatchSource:0}: Error finding container 25606cf2916dc2a1c922aceed5eab7be07fcd0db5ca5b1c17a14883b050d3ce2: Status 404 returned error can't find the container with id 25606cf2916dc2a1c922aceed5eab7be07fcd0db5ca5b1c17a14883b050d3ce2 Nov 01 21:49:02.585910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:02.585872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:02.953905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:02.953828 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" event={"ID":"fe02d9de-6509-43ff-a453-8f9373c24da8","Type":"ContainerStarted","Data":"25606cf2916dc2a1c922aceed5eab7be07fcd0db5ca5b1c17a14883b050d3ce2"} Nov 01 21:49:03.252966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:03.252887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:03.659286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:03.659251 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:03.791545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:03.791427 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:04.965266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:04.965179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" event={"ID":"fe02d9de-6509-43ff-a453-8f9373c24da8","Type":"ContainerStarted","Data":"fa3c0a8c88d200d37236b6e327fa69e65c80af9c775b6e69a199c5fb65290bc5"} Nov 01 21:49:04.965651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:04.965263 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:04.986661 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:04.986616 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" podStartSLOduration=1.4155011769999999 podStartE2EDuration="3.986603985s" totalImagesPullingTime="2.571102808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:04.984176163 +0000 UTC m=+298.677156649" watchObservedRunningTime="2026-11-01 21:49:04.986603985 +0000 UTC m=+298.679584472" Nov 01 21:49:06.466717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:06.466690 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:49:06.466717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:06.466713 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:49:07.264398 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.264365 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 21:49:07.269333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.269314 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.285752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.285729 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8grxv\"" Nov 01 21:49:07.286728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.286701 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 21:49:07.351532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.351506 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.351636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.351547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmnb7\" (UniqueName: \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.452788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.452762 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.452885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.452804 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnb7\" (UniqueName: \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.453122 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.453104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.468829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.468804 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnb7\" (UniqueName: \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\") pod \"kuadrant-operator-controller-manager-678d87766f-fhns5\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.581286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.581262 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:07.858912 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.858720 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 21:49:07.861742 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:07.861711 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f49525_ea81_41ec_8c2b_8def80641301.slice/crio-edbf016dc2aec5e3f9b09d527905ee609c67992a361660789e40c4e63dbc13f7 WatchSource:0}: Error finding container edbf016dc2aec5e3f9b09d527905ee609c67992a361660789e40c4e63dbc13f7: Status 404 returned error can't find the container with id edbf016dc2aec5e3f9b09d527905ee609c67992a361660789e40c4e63dbc13f7 Nov 01 21:49:07.928107 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.928086 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vd5qz" Nov 01 21:49:07.980501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:07.980470 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" event={"ID":"22f49525-ea81-41ec-8c2b-8def80641301","Type":"ContainerStarted","Data":"edbf016dc2aec5e3f9b09d527905ee609c67992a361660789e40c4e63dbc13f7"} Nov 01 21:49:08.798903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:08.798859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:08.932031 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:08.931992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:10.946239 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:10.946203 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvd26" Nov 01 21:49:11.815790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:11.815753 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.020672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.020628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.177708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.177636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.316510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.314900 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p"] Nov 01 21:49:12.318951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.318925 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.336814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.336789 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p"] Nov 01 21:49:12.338469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.338434 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gnpjh\"" Nov 01 21:49:12.345462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.345414 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.402046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402014 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssw8\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-kube-api-access-2ssw8\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402050 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36d41c-e520-4240-88fa-19e3d4f9386f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402403 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402522 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.402601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.402558 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.490555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.490476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.503655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503671 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503702 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503766 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503791 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503843 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssw8\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-kube-api-access-2ssw8\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503861 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36d41c-e520-4240-88fa-19e3d4f9386f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.503966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.503889 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.504098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.504014 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.504264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.504246 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.504387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.504363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.504920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.504895 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.505187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.505163 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36d41c-e520-4240-88fa-19e3d4f9386f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.506708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.506682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.507175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.507154 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.519927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.519903 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssw8\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-kube-api-access-2ssw8\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.520033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.519979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36d41c-e520-4240-88fa-19e3d4f9386f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886ft4x6p\" (UID: \"ca36d41c-e520-4240-88fa-19e3d4f9386f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.627924 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.627887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.634496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.634468 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:12.948856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.948825 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p"] Nov 01 21:49:12.950355 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:12.950330 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca36d41c_e520_4240_88fa_19e3d4f9386f.slice/crio-2e77d8b5b6bfb56b4677a0043a839a1003d0bdf4503d7568bdf6b87e73e3d75e WatchSource:0}: Error finding container 2e77d8b5b6bfb56b4677a0043a839a1003d0bdf4503d7568bdf6b87e73e3d75e: Status 404 returned error can't find the container with id 2e77d8b5b6bfb56b4677a0043a839a1003d0bdf4503d7568bdf6b87e73e3d75e Nov 01 21:49:12.957840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.957810 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:49:12.957932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.957886 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:49:12.957994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:12.957935 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:49:13.008778 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:13.008752 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" event={"ID":"ca36d41c-e520-4240-88fa-19e3d4f9386f","Type":"ContainerStarted","Data":"2e77d8b5b6bfb56b4677a0043a839a1003d0bdf4503d7568bdf6b87e73e3d75e"} Nov 01 21:49:13.445609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:13.445571 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:13.825433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:13.825390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:13.967011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:13.966976 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.016206 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.016159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" event={"ID":"ca36d41c-e520-4240-88fa-19e3d4f9386f","Type":"ContainerStarted","Data":"7b8288b9205762834a654f7db992f9e1c427f9d1a1ab3c7dfe68401311b186c2"} Nov 01 21:49:14.018626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.018595 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" event={"ID":"22f49525-ea81-41ec-8c2b-8def80641301","Type":"ContainerStarted","Data":"6575cdb9f445056fd481e3b5671f51dc5c0564187cdcc759e91abb262e436400"} Nov 01 21:49:14.018773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.018687 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:14.040573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.040529 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" podStartSLOduration=2.040515619 podStartE2EDuration="2.040515619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:14.038517611 +0000 UTC m=+307.731498100" watchObservedRunningTime="2026-11-01 21:49:14.040515619 +0000 UTC m=+307.733496104" Nov 01 21:49:14.061220 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.061173 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" podStartSLOduration=1.887825906 podStartE2EDuration="7.06116445s" totalImagesPullingTime="5.173338544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:14.060009619 +0000 UTC m=+307.752990108" watchObservedRunningTime="2026-11-01 21:49:14.06116445 +0000 UTC m=+307.754144935" Nov 01 21:49:14.093035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.092974 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.245012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.244984 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.274710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.274672 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:49:14.332247 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.332216 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:49:14.332247 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.332254 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:49:14.332557 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.332347 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.341244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.341219 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6fqfs\"" Nov 01 21:49:14.371267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.371207 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 21:49:14.422425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.422396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9sf\" (UniqueName: \"kubernetes.io/projected/d2808eb4-cd97-4607-a8db-8044d728df3d-kube-api-access-hc9sf\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.422586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.422527 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2808eb4-cd97-4607-a8db-8044d728df3d-config-file\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.443174 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.443145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.523221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.523186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9sf\" (UniqueName: \"kubernetes.io/projected/d2808eb4-cd97-4607-a8db-8044d728df3d-kube-api-access-hc9sf\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.523695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.523351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2808eb4-cd97-4607-a8db-8044d728df3d-config-file\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.524117 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.524094 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2808eb4-cd97-4607-a8db-8044d728df3d-config-file\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.542000 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.541970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9sf\" (UniqueName: \"kubernetes.io/projected/d2808eb4-cd97-4607-a8db-8044d728df3d-kube-api-access-hc9sf\") pod \"limitador-limitador-7c9b87d574-w2grp\" (UID: \"d2808eb4-cd97-4607-a8db-8044d728df3d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.635704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.635638 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:14.640736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.640711 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:14.646519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.646500 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:14.650357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.650303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.814292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.814254 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:14.818105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.818083 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:14.829823 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.829802 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmvjn\"" Nov 01 21:49:14.832127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.832107 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:14.904391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.904314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:14.921881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.921863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:49:14.923526 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:14.923504 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2808eb4_cd97_4607_a8db_8044d728df3d.slice/crio-3eb40767274bfd1b874dbe4e27212f0fea528042522f8cc5c45f7d7cb38873c0 WatchSource:0}: Error finding container 3eb40767274bfd1b874dbe4e27212f0fea528042522f8cc5c45f7d7cb38873c0: Status 404 returned error can't find the container with id 3eb40767274bfd1b874dbe4e27212f0fea528042522f8cc5c45f7d7cb38873c0 Nov 01 21:49:14.925914 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.925898 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:49:14.926040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:14.926012 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kj5\" (UniqueName: \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\") pod \"authorino-57f7f8659f-tkvkj\" (UID: \"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73\") " pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:15.025089 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.025059 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" event={"ID":"d2808eb4-cd97-4607-a8db-8044d728df3d","Type":"ContainerStarted","Data":"3eb40767274bfd1b874dbe4e27212f0fea528042522f8cc5c45f7d7cb38873c0"} Nov 01 21:49:15.025472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.025427 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:15.026203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.026183 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886ft4x6p" Nov 01 21:49:15.026436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.026415 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kj5\" (UniqueName: \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\") pod \"authorino-57f7f8659f-tkvkj\" (UID: \"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73\") " pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:15.035277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.035253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:15.038368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.038347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kj5\" (UniqueName: \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\") pod \"authorino-57f7f8659f-tkvkj\" (UID: \"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73\") " pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:15.139320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.139291 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:15.417563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.417536 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:15.419108 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:15.419082 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa5eec09_9ce4_429d_99f1_2f5ab27c2a73.slice/crio-b20e22e0250951aeb1f72b6af0127200772c5955963e16080fe2bd6ac593e588 WatchSource:0}: Error finding container b20e22e0250951aeb1f72b6af0127200772c5955963e16080fe2bd6ac593e588: Status 404 returned error can't find the container with id b20e22e0250951aeb1f72b6af0127200772c5955963e16080fe2bd6ac593e588 Nov 01 21:49:15.972820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:15.972792 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t29lh" Nov 01 21:49:16.032266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:16.032218 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" event={"ID":"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73","Type":"ContainerStarted","Data":"b20e22e0250951aeb1f72b6af0127200772c5955963e16080fe2bd6ac593e588"} Nov 01 21:49:16.469645 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:16.469609 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:16.604641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:16.604606 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:17.038405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:17.038368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" event={"ID":"d2808eb4-cd97-4607-a8db-8044d728df3d","Type":"ContainerStarted","Data":"9f7bf0f8c9f97340b4d1a657da6586a3b2d58a9b6f45bbff77723bc98ce6ee9c"} Nov 01 21:49:17.038816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:17.038711 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:17.070687 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:17.070643 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" podStartSLOduration=1.844104789 podStartE2EDuration="3.070629915s" totalImagesPullingTime="1.226525126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:17.068702804 +0000 UTC m=+310.761683290" watchObservedRunningTime="2026-11-01 21:49:17.070629915 +0000 UTC m=+310.763610401" Nov 01 21:49:18.045339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:18.045297 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" event={"ID":"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73","Type":"ContainerStarted","Data":"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd"} Nov 01 21:49:18.063501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:18.063417 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" podStartSLOduration=1.5632818099999999 podStartE2EDuration="4.063404821s" totalImagesPullingTime="2.500123011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:18.061422117 +0000 UTC m=+311.754402605" watchObservedRunningTime="2026-11-01 21:49:18.063404821 +0000 UTC m=+311.756385310" Nov 01 21:49:18.856315 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:49:18.856279 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:49:19.070728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:19.070695 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:20.060952 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:20.060918 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" podUID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" containerName="authorino" containerID="cri-o://f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd" gracePeriod=30 Nov 01 21:49:20.307337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:20.307309 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:20.370801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:20.370737 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\" (UniqueName: \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\") pod \"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73\" (UID: \"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73\") " Nov 01 21:49:20.372859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:20.372834 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5" pod "aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" (UID: "aa5eec09-9ce4-429d-99f1-2f5ab27c2a73"). InnerVolumeSpecName "kube-api-access-k7kj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:49:20.472143 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:20.472121 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kj5\" (UniqueName: \"kubernetes.io/projected/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73-kube-api-access-k7kj5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:49:21.066608 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.066576 5036 generic.go:356] "Generic (PLEG): container finished" podID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" containerID="f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd" exitCode=0 Nov 01 21:49:21.066789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.066619 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" Nov 01 21:49:21.066789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.066660 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" event={"ID":"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73","Type":"ContainerDied","Data":"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd"} Nov 01 21:49:21.066789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.066708 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tkvkj" event={"ID":"aa5eec09-9ce4-429d-99f1-2f5ab27c2a73","Type":"ContainerDied","Data":"b20e22e0250951aeb1f72b6af0127200772c5955963e16080fe2bd6ac593e588"} Nov 01 21:49:21.066789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.066726 5036 scope.go:122] "RemoveContainer" containerID="f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd" Nov 01 21:49:21.075378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.075359 5036 scope.go:122] "RemoveContainer" containerID="f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd" Nov 01 21:49:21.075647 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:49:21.075629 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd\": container with ID starting with f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd not found: ID does not exist" containerID="f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd" Nov 01 21:49:21.075729 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.075654 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd"} err="failed to get container status \"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd\": rpc error: code = NotFound desc = could not find container \"f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd\": container with ID starting with f3a104a9dc04c961ca81bebeadbaaf8027e9568abacea5d488a31f60688f8abd not found: ID does not exist" Nov 01 21:49:21.085516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.085488 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:21.089365 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.089345 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tkvkj"] Nov 01 21:49:21.223841 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.223813 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:21.355753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:21.355672 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:22.436535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:22.436500 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" path="/var/lib/kubelet/pods/aa5eec09-9ce4-429d-99f1-2f5ab27c2a73/volumes" Nov 01 21:49:25.028131 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:25.028094 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 21:49:27.863044 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:27.863012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:27.990500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:27.990432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:28.046272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:28.046246 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2grp" Nov 01 21:49:29.166570 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:29.166535 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:29.300169 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:29.300133 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:29.917888 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:29.917853 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:33.623614 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:33.623571 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:33.755192 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:33.755153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:34.062556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:34.062523 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:34.182876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:34.182838 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:37.475099 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:37.475066 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:37.599747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:37.599716 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:41.268569 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:41.268533 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:41.403040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:41.403001 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:48.855971 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:49:48.855931 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:49:49.350582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:49.350547 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:49.475387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:49.475331 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:49.619873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:49.619789 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:49.755666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:49.755629 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:53.495955 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:53.661117 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:53.661078 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:54.409272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.409241 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l"] Nov 01 21:49:54.409847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.409828 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" containerName="authorino" Nov 01 21:49:54.410019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.409998 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" containerName="authorino" Nov 01 21:49:54.410183 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.410144 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5eec09-9ce4-429d-99f1-2f5ab27c2a73" containerName="authorino" Nov 01 21:49:54.415933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.415915 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.421731 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.421710 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l"] Nov 01 21:49:54.422642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.422604 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 21:49:54.423741 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.423722 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 21:49:54.423975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.423916 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 21:49:54.423975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.423967 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 21:49:54.457362 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.457333 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 21:49:54.457362 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.457343 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-5t57l\"" Nov 01 21:49:54.493844 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.493817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:54.542734 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.542704 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.543118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.542758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e20c269e-61ab-4af3-b1bf-eba3a120df3a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.543118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.542881 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.543118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.542904 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2jl\" (UniqueName: \"kubernetes.io/projected/e20c269e-61ab-4af3-b1bf-eba3a120df3a-kube-api-access-4x2jl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.620607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.620579 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:54.643760 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.643735 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e20c269e-61ab-4af3-b1bf-eba3a120df3a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.643851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.643774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.643851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.643794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2jl\" (UniqueName: \"kubernetes.io/projected/e20c269e-61ab-4af3-b1bf-eba3a120df3a-kube-api-access-4x2jl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.643924 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.643854 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.644644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.644617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.644854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.644837 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e20c269e-61ab-4af3-b1bf-eba3a120df3a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.646244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.646225 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e20c269e-61ab-4af3-b1bf-eba3a120df3a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.656219 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.656191 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2jl\" (UniqueName: \"kubernetes.io/projected/e20c269e-61ab-4af3-b1bf-eba3a120df3a-kube-api-access-4x2jl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mrv7l\" (UID: \"e20c269e-61ab-4af3-b1bf-eba3a120df3a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:54.734061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:54.734011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:49:55.008967 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:55.008943 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l"] Nov 01 21:49:55.010731 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:49:55.010699 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20c269e_61ab_4af3_b1bf_eba3a120df3a.slice/crio-ceebc7dd2cc4ce7ccc7f54c1b919ea8e4663b0572e0adbfb8432ac720653165a WatchSource:0}: Error finding container ceebc7dd2cc4ce7ccc7f54c1b919ea8e4663b0572e0adbfb8432ac720653165a: Status 404 returned error can't find the container with id ceebc7dd2cc4ce7ccc7f54c1b919ea8e4663b0572e0adbfb8432ac720653165a Nov 01 21:49:55.243422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:55.243382 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" event={"ID":"e20c269e-61ab-4af3-b1bf-eba3a120df3a","Type":"ContainerStarted","Data":"ceebc7dd2cc4ce7ccc7f54c1b919ea8e4663b0572e0adbfb8432ac720653165a"} Nov 01 21:49:55.956609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:49:55.956576 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:06.694657 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.694618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:06.781577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.781542 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:50:06.788922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.788897 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.795649 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.795622 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:50:06.805869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.805582 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 21:50:06.805869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.805641 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-498mr\"" Nov 01 21:50:06.829575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.829544 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zfclj"] Nov 01 21:50:06.835003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.834983 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.844100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.843959 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 21:50:06.844100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.844058 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 21:50:06.844100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.844076 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-snmf2\"" Nov 01 21:50:06.846706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.846606 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:06.848456 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.848382 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zfclj"] Nov 01 21:50:06.862508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.862478 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.862609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.862525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.862609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.862567 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6msq\" (UniqueName: \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.862726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.862649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-metrics-certs\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.862726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.862692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbxn\" (UniqueName: \"kubernetes.io/projected/31186970-bfbd-4a9d-8923-ddb8582ff2e4-kube-api-access-tsbxn\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.964010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.963972 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.964010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.964010 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.964268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.964034 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6msq\" (UniqueName: \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.964268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.964094 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-metrics-certs\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.964268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.964129 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbxn\" (UniqueName: \"kubernetes.io/projected/31186970-bfbd-4a9d-8923-ddb8582ff2e4-kube-api-access-tsbxn\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.964268 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:06.964136 5036 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Nov 01 21:50:06.964268 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:06.964208 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert podName:31186970-bfbd-4a9d-8923-ddb8582ff2e4 nodeName:}" failed. No retries permitted until 2026-11-01 21:50:07.464186745 +0000 UTC m=+361.157167212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert") pod "odh-model-controller-769875db56-zfclj" (UID: "31186970-bfbd-4a9d-8923-ddb8582ff2e4") : secret "odh-model-controller-webhook-cert" not found Nov 01 21:50:06.967309 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.967285 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-metrics-certs\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:06.967417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.967285 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.974892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.974835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:06.977671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.977130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6msq\" (UniqueName: \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\") pod \"llmisvc-controller-manager-869d5dcc5d-6c6t4\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:06.977671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:06.977608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbxn\" (UniqueName: \"kubernetes.io/projected/31186970-bfbd-4a9d-8923-ddb8582ff2e4-kube-api-access-tsbxn\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:07.106651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.106561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:07.107563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.107540 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:07.236889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.236845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:07.403530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.403438 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:50:07.407816 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:50:07.407776 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e3dd292_b942_4e3c_8c77_3ae0992daeb8.slice/crio-a573f9a46baed79543e6312a550d47fa0b70d0d66c7b3fe69137039bd5540ae7 WatchSource:0}: Error finding container a573f9a46baed79543e6312a550d47fa0b70d0d66c7b3fe69137039bd5540ae7: Status 404 returned error can't find the container with id a573f9a46baed79543e6312a550d47fa0b70d0d66c7b3fe69137039bd5540ae7 Nov 01 21:50:07.470031 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.469998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:07.472689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.472660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31186970-bfbd-4a9d-8923-ddb8582ff2e4-cert\") pod \"odh-model-controller-769875db56-zfclj\" (UID: \"31186970-bfbd-4a9d-8923-ddb8582ff2e4\") " pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:07.754335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.754254 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:07.867297 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:07.865978 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:08.060242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.060206 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zfclj"] Nov 01 21:50:08.062247 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:50:08.062202 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31186970_bfbd_4a9d_8923_ddb8582ff2e4.slice/crio-5ad8962022eaf9579b49b9b92691e686ed1803223487da1c0700ecc6c278fa7f WatchSource:0}: Error finding container 5ad8962022eaf9579b49b9b92691e686ed1803223487da1c0700ecc6c278fa7f: Status 404 returned error can't find the container with id 5ad8962022eaf9579b49b9b92691e686ed1803223487da1c0700ecc6c278fa7f Nov 01 21:50:08.075768 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.075737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:08.201124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.201002 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:08.322351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.322280 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" event={"ID":"4e3dd292-b942-4e3c-8c77-3ae0992daeb8","Type":"ContainerStarted","Data":"a573f9a46baed79543e6312a550d47fa0b70d0d66c7b3fe69137039bd5540ae7"} Nov 01 21:50:08.324999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.324951 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zfclj" event={"ID":"31186970-bfbd-4a9d-8923-ddb8582ff2e4","Type":"ContainerStarted","Data":"5ad8962022eaf9579b49b9b92691e686ed1803223487da1c0700ecc6c278fa7f"} Nov 01 21:50:08.340311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.340279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:08.468044 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:08.468011 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:09.336584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:09.336509 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" event={"ID":"e20c269e-61ab-4af3-b1bf-eba3a120df3a","Type":"ContainerStarted","Data":"e4667e924c800e5cd7eaae6bfb5487fddc90c8a4770569d6464d0c5bfe2ea7cc"} Nov 01 21:50:09.337060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:09.336684 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:50:09.358273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:09.358213 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" podStartSLOduration=2.074069156 podStartE2EDuration="15.358194086s" totalImagesPullingTime="13.28412493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:09.356625681 +0000 UTC m=+363.049606168" watchObservedRunningTime="2026-11-01 21:50:09.358194086 +0000 UTC m=+363.051174573" Nov 01 21:50:11.349874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.349837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" event={"ID":"4e3dd292-b942-4e3c-8c77-3ae0992daeb8","Type":"ContainerStarted","Data":"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d"} Nov 01 21:50:11.350301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.350022 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:11.351366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.351342 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zfclj" event={"ID":"31186970-bfbd-4a9d-8923-ddb8582ff2e4","Type":"ContainerStarted","Data":"b4a29813d2c4745c112d7a1ceb8d6f9ad323835c9ad7748f3d4946608b5e2cf6"} Nov 01 21:50:11.351517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.351498 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:11.369259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.369218 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" podStartSLOduration=2.300548637 podStartE2EDuration="5.369205786s" totalImagesPullingTime="3.068657149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:11.368909592 +0000 UTC m=+365.061890078" watchObservedRunningTime="2026-11-01 21:50:11.369205786 +0000 UTC m=+365.062186302" Nov 01 21:50:11.389293 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:11.389227 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zfclj" podStartSLOduration=2.530510843 podStartE2EDuration="5.389216161s" totalImagesPullingTime="2.858705318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:11.388176996 +0000 UTC m=+365.081157484" watchObservedRunningTime="2026-11-01 21:50:11.389216161 +0000 UTC m=+365.082196647" Nov 01 21:50:18.856555 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:18.856514 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:50:20.348653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:20.348623 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" Nov 01 21:50:22.358722 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:22.358688 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zfclj" Nov 01 21:50:23.711956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:23.711919 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:23.843785 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:23.843748 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:25.017161 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:25.017126 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:25.151003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:25.150972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:25.275826 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:25.275798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:25.867562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:25.867521 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:50:33.662594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:33.662554 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:50:33.662975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:33.662619 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:50:34.163812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.163767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:34.329560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.329521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:34.412202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.412170 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-m4frg"] Nov 01 21:50:34.924238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.924205 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-m4frg"] Nov 01 21:50:34.924730 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.924290 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-m4frg"] Nov 01 21:50:34.924730 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.924324 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:34.924898 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:34.924875 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rbt4z tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-rbt4z tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-m4frg" podUID="c77e1888-c3a6-4dc7-8986-fe4519b509f2" Nov 01 21:50:34.930603 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.930583 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6wkd"] Nov 01 21:50:34.960042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.960011 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6wkd"] Nov 01 21:50:34.960156 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.960058 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6wkd"] Nov 01 21:50:34.960156 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.960080 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:50:34.960255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.960233 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:34.960852 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:34.960820 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9d2v4 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-9d2v4 tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" podUID="457ab01a-7c69-46c4-91a0-4c91b799e06f" Nov 01 21:50:34.992073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.992053 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:50:34.992181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.992129 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:34.996228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.996206 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 21:50:34.996328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:34.996209 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmvjn\"" Nov 01 21:50:35.008494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.008433 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2v4\" (UniqueName: \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.008599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.008577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.008662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.008613 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbt4z\" (UniqueName: \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.008662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.008651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109783 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllwn\" (UniqueName: \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109812 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2v4\" (UniqueName: \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109844 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109864 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbt4z\" (UniqueName: \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.109970 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.112093 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.112181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.122659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbt4z\" (UniqueName: \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\") pod \"authorino-6dd76875b5-m4frg\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.131058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.122925 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2v4\" (UniqueName: \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\") pod \"authorino-6889cbd8c4-n6wkd\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.299582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.210533 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.299582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.210634 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllwn\" (UniqueName: \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.299582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.212660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.299582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.222836 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllwn\" (UniqueName: \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\") pod \"authorino-69f9c8b9c5-knsms\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.347077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.347025 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:50:35.476098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.476069 5036 generic.go:356] "Generic (PLEG): container finished" podID="83870513-6b8e-4205-9823-87b037b374b8" containerID="aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8" exitCode=0 Nov 01 21:50:35.476228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.476136 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerDied","Data":"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8"} Nov 01 21:50:35.476228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.476157 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.476228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.476162 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.476528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.476506 5036 scope.go:122] "RemoveContainer" containerID="aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8" Nov 01 21:50:35.477787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.477762 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:50:35.477854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.477828 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:50:35.477890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.477860 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:50:35.512890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.512871 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:35.531542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.531518 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:35.613742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.613667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\" (UniqueName: \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\") pod \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " Nov 01 21:50:35.613742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.613736 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\" (UniqueName: \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\") pod \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\" (UID: \"c77e1888-c3a6-4dc7-8986-fe4519b509f2\") " Nov 01 21:50:35.613954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.613782 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\" (UniqueName: \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\") pod \"457ab01a-7c69-46c4-91a0-4c91b799e06f\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " Nov 01 21:50:35.613954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.613812 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\" (UniqueName: \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\") pod \"457ab01a-7c69-46c4-91a0-4c91b799e06f\" (UID: \"457ab01a-7c69-46c4-91a0-4c91b799e06f\") " Nov 01 21:50:35.617162 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.617128 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert" pod "c77e1888-c3a6-4dc7-8986-fe4519b509f2" (UID: "c77e1888-c3a6-4dc7-8986-fe4519b509f2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:35.617162 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.617143 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z" pod "c77e1888-c3a6-4dc7-8986-fe4519b509f2" (UID: "c77e1888-c3a6-4dc7-8986-fe4519b509f2"). InnerVolumeSpecName "kube-api-access-rbt4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:35.617377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.617223 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4" pod "457ab01a-7c69-46c4-91a0-4c91b799e06f" (UID: "457ab01a-7c69-46c4-91a0-4c91b799e06f"). InnerVolumeSpecName "kube-api-access-9d2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:35.617377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.617370 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert" pod "457ab01a-7c69-46c4-91a0-4c91b799e06f" (UID: "457ab01a-7c69-46c4-91a0-4c91b799e06f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:35.633052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.633027 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:50:35.635004 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:50:35.634973 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81a4003d_a49e_4a0e_8819_c0b766d747bf.slice/crio-ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e WatchSource:0}: Error finding container ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e: Status 404 returned error can't find the container with id ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e Nov 01 21:50:35.715581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.715557 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbt4z\" (UniqueName: \"kubernetes.io/projected/c77e1888-c3a6-4dc7-8986-fe4519b509f2-kube-api-access-rbt4z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:35.715581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.715578 5036 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c77e1888-c3a6-4dc7-8986-fe4519b509f2-tls-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:35.715581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.715588 5036 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457ab01a-7c69-46c4-91a0-4c91b799e06f-tls-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:35.715777 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:35.715597 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2v4\" (UniqueName: \"kubernetes.io/projected/457ab01a-7c69-46c4-91a0-4c91b799e06f-kube-api-access-9d2v4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:36.506172 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.506088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" event={"ID":"81a4003d-a49e-4a0e-8819-c0b766d747bf","Type":"ContainerStarted","Data":"ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e"} Nov 01 21:50:36.508030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.508008 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-m4frg" Nov 01 21:50:36.508217 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.508023 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerStarted","Data":"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f"} Nov 01 21:50:36.508287 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.508220 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n6wkd" Nov 01 21:50:36.541577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.541552 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6wkd"] Nov 01 21:50:36.547725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.547700 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n6wkd"] Nov 01 21:50:36.614642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.614611 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-m4frg"] Nov 01 21:50:36.621810 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:36.621781 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-m4frg"] Nov 01 21:50:37.317571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:37.317535 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:50:37.317856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:37.317831 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 01 21:50:37.317939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:37.317892 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 01 21:50:37.514594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:37.514556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" event={"ID":"81a4003d-a49e-4a0e-8819-c0b766d747bf","Type":"ContainerStarted","Data":"411b942b017430c2ee9f5d699fd139cde477bddf84d6714d0d68acce09b001f7"} Nov 01 21:50:37.533136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:37.533080 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" podStartSLOduration=2.464552759 podStartE2EDuration="3.533065413s" totalImagesPullingTime="1.068512654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:37.532267138 +0000 UTC m=+391.225247624" watchObservedRunningTime="2026-11-01 21:50:37.533065413 +0000 UTC m=+391.226045914" Nov 01 21:50:38.317766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:38.317731 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 01 21:50:38.317946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:38.317790 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 01 21:50:38.436640 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:38.436609 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457ab01a-7c69-46c4-91a0-4c91b799e06f" path="/var/lib/kubelet/pods/457ab01a-7c69-46c4-91a0-4c91b799e06f/volumes" Nov 01 21:50:38.436883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:38.436871 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77e1888-c3a6-4dc7-8986-fe4519b509f2" path="/var/lib/kubelet/pods/c77e1888-c3a6-4dc7-8986-fe4519b509f2/volumes" Nov 01 21:50:39.317918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:39.317879 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 01 21:50:39.318324 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:39.317934 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 01 21:50:40.317773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:40.317735 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 01 21:50:40.317958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:40.317792 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 01 21:50:41.317824 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:41.317785 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 01 21:50:41.318006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:41.317841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 01 21:50:42.320979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:42.320952 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:50:42.321480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:42.321249 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:50:42.322002 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:42.321982 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 21:50:42.358019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:42.357997 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:50:48.856733 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:50:48.856653 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:50:56.017383 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:50:56.017344 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:18.781856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.781821 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pggxq"] Nov 01 21:51:18.785284 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.785267 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.795292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.795263 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pggxq"] Nov 01 21:51:18.856190 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:51:18.856155 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:51:18.883661 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.883636 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jn6d\" (UniqueName: \"kubernetes.io/projected/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-kube-api-access-5jn6d\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.883795 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.883686 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.984832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.984796 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jn6d\" (UniqueName: \"kubernetes.io/projected/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-kube-api-access-5jn6d\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.984983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.984846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.987194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.987174 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:18.997765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:18.997742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jn6d\" (UniqueName: \"kubernetes.io/projected/6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8-kube-api-access-5jn6d\") pod \"llmisvc-controller-manager-6466c554d8-pggxq\" (UID: \"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:19.099595 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:19.099535 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:19.372215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:19.372186 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pggxq"] Nov 01 21:51:19.374098 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:51:19.374074 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d80abf6_5dfc_4ea6_8a17_baa20b4da2c8.slice/crio-4c4d5c212345ef205900f0d171d6c3ebe201b6d840899abe523418e1111f1d0e WatchSource:0}: Error finding container 4c4d5c212345ef205900f0d171d6c3ebe201b6d840899abe523418e1111f1d0e: Status 404 returned error can't find the container with id 4c4d5c212345ef205900f0d171d6c3ebe201b6d840899abe523418e1111f1d0e Nov 01 21:51:19.725220 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:19.725142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" event={"ID":"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8","Type":"ContainerStarted","Data":"4c4d5c212345ef205900f0d171d6c3ebe201b6d840899abe523418e1111f1d0e"} Nov 01 21:51:20.732162 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:20.732125 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" event={"ID":"6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8","Type":"ContainerStarted","Data":"04c1e224d67a49eefe1d3ed73ced3dd639fc2e75ff1ad6ad7b98ed36079d7e51"} Nov 01 21:51:20.732666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:20.732246 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:20.752180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:20.752132 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" podStartSLOduration=2.367396388 podStartE2EDuration="2.752119675s" totalImagesPullingTime="384.723287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:51:20.751010792 +0000 UTC m=+434.443991278" watchObservedRunningTime="2026-11-01 21:51:20.752119675 +0000 UTC m=+434.445100160" Nov 01 21:51:47.372280 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:47.372238 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:47.504572 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:47.504540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:48.860216 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:51:48.860171 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:51:51.738210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:51.738182 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pggxq" Nov 01 21:51:51.784946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:51.784905 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:51:51.785546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:51.785212 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" podUID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" containerName="manager" containerID="cri-o://1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d" gracePeriod=30 Nov 01 21:51:52.041959 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.041935 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:51:52.147128 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.147100 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\" (UniqueName: \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\") pod \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " Nov 01 21:51:52.147255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.147204 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\" (UniqueName: \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\") pod \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\" (UID: \"4e3dd292-b942-4e3c-8c77-3ae0992daeb8\") " Nov 01 21:51:52.149278 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.149250 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq" pod "4e3dd292-b942-4e3c-8c77-3ae0992daeb8" (UID: "4e3dd292-b942-4e3c-8c77-3ae0992daeb8"). InnerVolumeSpecName "kube-api-access-z6msq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:51:52.149368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.149311 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert" pod "4e3dd292-b942-4e3c-8c77-3ae0992daeb8" (UID: "4e3dd292-b942-4e3c-8c77-3ae0992daeb8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:51:52.248364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.248338 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6msq\" (UniqueName: \"kubernetes.io/projected/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-kube-api-access-z6msq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:51:52.248364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.248361 5036 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e3dd292-b942-4e3c-8c77-3ae0992daeb8-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:51:52.642370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.642332 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:52.777097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.777065 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:52.895302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.895275 5036 generic.go:356] "Generic (PLEG): container finished" podID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" containerID="1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d" exitCode=0 Nov 01 21:51:52.895477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.895317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" event={"ID":"4e3dd292-b942-4e3c-8c77-3ae0992daeb8","Type":"ContainerDied","Data":"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d"} Nov 01 21:51:52.895477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.895333 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" Nov 01 21:51:52.895477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.895357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4" event={"ID":"4e3dd292-b942-4e3c-8c77-3ae0992daeb8","Type":"ContainerDied","Data":"a573f9a46baed79543e6312a550d47fa0b70d0d66c7b3fe69137039bd5540ae7"} Nov 01 21:51:52.895477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.895379 5036 scope.go:122] "RemoveContainer" containerID="1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d" Nov 01 21:51:52.904282 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.904260 5036 scope.go:122] "RemoveContainer" containerID="1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d" Nov 01 21:51:52.904584 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:51:52.904558 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d\": container with ID starting with 1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d not found: ID does not exist" containerID="1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d" Nov 01 21:51:52.904653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.904598 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d"} err="failed to get container status \"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d\": rpc error: code = NotFound desc = could not find container \"1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d\": container with ID starting with 1cf924ecfce44421c85abe04c7b15e1c283684fe4ac7e4f4583bdfa2e5dc828d not found: ID does not exist" Nov 01 21:51:52.914595 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.914570 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:51:52.918575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:52.918554 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6c6t4"] Nov 01 21:51:54.437756 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:54.437716 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" path="/var/lib/kubelet/pods/4e3dd292-b942-4e3c-8c77-3ae0992daeb8/volumes" Nov 01 21:51:56.071238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:51:56.071206 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:11.294747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:11.294707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:11.692124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:11.692044 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:13.294928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:13.294892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:13.714646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:13.714568 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:14.159466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:14.159418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:15.009698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:15.009646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:15.969477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:15.969420 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:16.234559 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:16.234477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:17.745762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:17.745625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:18.347993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:18.347950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:18.465408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:18.465359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:18.782498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:18.782430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:18.864566 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:52:18.864529 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:52:19.693872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:19.693834 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:19.965321 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:19.965228 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:20.090555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:20.090519 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:21.690095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:21.690060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:24.772825 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:24.772789 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:44.484107 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:44.484070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:52:44.955881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:44.955841 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:52:44.956055 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:44.956027 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" podUID="81a4003d-a49e-4a0e-8819-c0b766d747bf" containerName="authorino" containerID="cri-o://411b942b017430c2ee9f5d699fd139cde477bddf84d6714d0d68acce09b001f7" gracePeriod=30 Nov 01 21:52:45.213388 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.213318 5036 generic.go:356] "Generic (PLEG): container finished" podID="81a4003d-a49e-4a0e-8819-c0b766d747bf" containerID="411b942b017430c2ee9f5d699fd139cde477bddf84d6714d0d68acce09b001f7" exitCode=0 Nov 01 21:52:45.213584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.213389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" event={"ID":"81a4003d-a49e-4a0e-8819-c0b766d747bf","Type":"ContainerDied","Data":"411b942b017430c2ee9f5d699fd139cde477bddf84d6714d0d68acce09b001f7"} Nov 01 21:52:45.213584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.213470 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" event={"ID":"81a4003d-a49e-4a0e-8819-c0b766d747bf","Type":"ContainerDied","Data":"ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e"} Nov 01 21:52:45.213584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.213486 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd06f12be32f028e9f7128f39f71cdb19151499890198f6c90f9d2682a82b1e" Nov 01 21:52:45.214500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.214482 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:52:45.330983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.330946 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\" (UniqueName: \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\") pod \"81a4003d-a49e-4a0e-8819-c0b766d747bf\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " Nov 01 21:52:45.331154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.331051 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\" (UniqueName: \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\") pod \"81a4003d-a49e-4a0e-8819-c0b766d747bf\" (UID: \"81a4003d-a49e-4a0e-8819-c0b766d747bf\") " Nov 01 21:52:45.333154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.333122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn" pod "81a4003d-a49e-4a0e-8819-c0b766d747bf" (UID: "81a4003d-a49e-4a0e-8819-c0b766d747bf"). InnerVolumeSpecName "kube-api-access-wllwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:45.342415 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.342384 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert" pod "81a4003d-a49e-4a0e-8819-c0b766d747bf" (UID: "81a4003d-a49e-4a0e-8819-c0b766d747bf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:52:45.432385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.432349 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wllwn\" (UniqueName: \"kubernetes.io/projected/81a4003d-a49e-4a0e-8819-c0b766d747bf-kube-api-access-wllwn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:52:45.432385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.432381 5036 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a4003d-a49e-4a0e-8819-c0b766d747bf-tls-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:52:45.558712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.558678 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:45.686917 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:45.686877 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:46.221787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:46.221740 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f9c8b9c5-knsms" Nov 01 21:52:46.303457 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:46.303411 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:52:46.327976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:46.327943 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69f9c8b9c5-knsms"] Nov 01 21:52:46.437851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:46.437783 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a4003d-a49e-4a0e-8819-c0b766d747bf" path="/var/lib/kubelet/pods/81a4003d-a49e-4a0e-8819-c0b766d747bf/volumes" Nov 01 21:52:48.857180 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:52:48.857145 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:52:52.449337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:52.449301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:52.590805 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:52.590770 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:56.116046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:52:56.116003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:07.401101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401067 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz"] Nov 01 21:53:07.401708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401677 5036 cpu_manager.go:414] "containerMap: removing container" podUID="81a4003d-a49e-4a0e-8819-c0b766d747bf" containerName="authorino" Nov 01 21:53:07.401708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401700 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a4003d-a49e-4a0e-8819-c0b766d747bf" containerName="authorino" Nov 01 21:53:07.401920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401723 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" containerName="manager" Nov 01 21:53:07.401920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401730 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" containerName="manager" Nov 01 21:53:07.401920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401869 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a4003d-a49e-4a0e-8819-c0b766d747bf" containerName="authorino" Nov 01 21:53:07.401920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.401887 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3dd292-b942-4e3c-8c77-3ae0992daeb8" containerName="manager" Nov 01 21:53:07.405422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.405403 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.419224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.419201 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz"] Nov 01 21:53:07.520245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520217 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grqd\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-kube-api-access-2grqd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520248 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520275 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520347 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520545 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61880c-597e-4979-bda6-9f24361aaabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.520799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.520670 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61880c-597e-4979-bda6-9f24361aaabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621763 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621732 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621771 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.621913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61880c-597e-4979-bda6-9f24361aaabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622174 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.621931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61880c-597e-4979-bda6-9f24361aaabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622174 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622035 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grqd\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-kube-api-access-2grqd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622174 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622287 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622212 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622287 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622367 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622583 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622564 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.622916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.622896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61880c-597e-4979-bda6-9f24361aaabd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.624178 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.624158 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61880c-597e-4979-bda6-9f24361aaabd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.624406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.624386 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61880c-597e-4979-bda6-9f24361aaabd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.634405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.634376 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grqd\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-kube-api-access-2grqd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.634405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.634393 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61880c-597e-4979-bda6-9f24361aaabd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m9bdz\" (UID: \"fb61880c-597e-4979-bda6-9f24361aaabd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:07.720062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:07.720011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:08.001514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.001487 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz"] Nov 01 21:53:08.003226 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:08.003196 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb61880c_597e_4979_bda6_9f24361aaabd.slice/crio-9f20bfe04c35c3486f9f635fe0ba1d69a5de6d6b8ea8dcd24937194bef42a087 WatchSource:0}: Error finding container 9f20bfe04c35c3486f9f635fe0ba1d69a5de6d6b8ea8dcd24937194bef42a087: Status 404 returned error can't find the container with id 9f20bfe04c35c3486f9f635fe0ba1d69a5de6d6b8ea8dcd24937194bef42a087 Nov 01 21:53:08.005215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.005184 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:08.005311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.005245 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:08.005311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.005271 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:08.345700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.345628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" event={"ID":"fb61880c-597e-4979-bda6-9f24361aaabd","Type":"ContainerStarted","Data":"ac7551cda765b380d5606112acf9d4ab0152cd283eafa1262c66720442454fbd"} Nov 01 21:53:08.346118 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.345754 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" event={"ID":"fb61880c-597e-4979-bda6-9f24361aaabd","Type":"ContainerStarted","Data":"9f20bfe04c35c3486f9f635fe0ba1d69a5de6d6b8ea8dcd24937194bef42a087"} Nov 01 21:53:08.372264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.372215 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podStartSLOduration=1.372201875 podStartE2EDuration="1.372201875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:08.369549851 +0000 UTC m=+542.062530337" watchObservedRunningTime="2026-11-01 21:53:08.372201875 +0000 UTC m=+542.065182361" Nov 01 21:53:08.695522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.695106 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:08.720755 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:08.720728 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:09.721785 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:09.721751 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:09.722166 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:09.721801 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:10.721101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:10.721063 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:10.721267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:10.721122 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:11.721148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:11.721110 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:11.721590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:11.721168 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:12.720552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:12.720517 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:12.720735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:12.720569 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:13.720685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:13.720641 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:13.721204 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:13.720706 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:14.721145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:14.721109 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 01 21:53:14.721554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:14.721168 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 01 21:53:16.148767 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:16.148734 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:16.149232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:16.149202 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:16.150078 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:16.150059 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" Nov 01 21:53:17.077212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:17.077176 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:17.540076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:17.540028 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:18.856436 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:53:18.856399 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:53:19.047885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:19.047847 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:19.171798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:19.171712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:20.515507 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:20.515467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:20.639469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:20.639416 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:22.415401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.415357 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 21:53:22.419384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.419354 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.438136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.438111 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 21:53:22.472999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.472970 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473188 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473141 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473321 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473302 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473363 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2v9j\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473426 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473405 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473477 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.473556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.473523 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574332 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574377 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574400 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2v9j\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574463 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574609 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574649 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574675 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574727 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574845 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.574990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.574908 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.575097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.575002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.575483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.575440 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.576814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.576789 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.577218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.577201 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.587127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.587103 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.587208 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.587150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2v9j\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9sjh9\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:22.734622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:22.734554 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:23.015713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.015686 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 21:53:23.016636 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:23.016612 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9285de_c1a7_419e_bbb9_c083a10cf300.slice/crio-b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463 WatchSource:0}: Error finding container b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463: Status 404 returned error can't find the container with id b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463 Nov 01 21:53:23.018863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.018831 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:23.018947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.018898 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:23.018947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.018933 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:53:23.430208 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.430172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" event={"ID":"5d9285de-c1a7-419e-bbb9-c083a10cf300","Type":"ContainerStarted","Data":"1396471fc8d520253932413bfeb99dac66090a27504d4588ef8171abd952b571"} Nov 01 21:53:23.430618 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.430233 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" event={"ID":"5d9285de-c1a7-419e-bbb9-c083a10cf300","Type":"ContainerStarted","Data":"b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463"} Nov 01 21:53:23.458103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.458054 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podStartSLOduration=1.458041298 podStartE2EDuration="1.458041298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:23.456769501 +0000 UTC m=+557.149749985" watchObservedRunningTime="2026-11-01 21:53:23.458041298 +0000 UTC m=+557.151021781" Nov 01 21:53:23.735678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:23.735587 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:24.736373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:24.736331 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:24.736772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:24.736399 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:25.735218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:25.735183 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:25.735380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:25.735235 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:26.735360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:26.735327 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:26.735742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:26.735374 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:27.735622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:27.735588 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:27.736108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:27.735648 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:28.735597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:28.735558 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:28.735784 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:28.735616 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:29.735262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:29.735221 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 21:53:29.735498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:29.735303 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 21:53:31.171670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:31.171635 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:31.172116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:31.172051 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:31.172764 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:31.172739 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 21:53:46.970637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:46.970598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:48.870894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:48.870801 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:48.872854 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:53:48.872818 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:53:50.711813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:50.711773 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:50.862283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:50.862231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:50.993653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:50.993536 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:51.129931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:51.129891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:51.267621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:51.267532 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:51.429702 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:51.429666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:51.720909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:51.720873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:51.940898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:51.940862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:52.146887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.146848 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:52.191016 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.190984 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 21:53:52.195263 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.195240 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.205479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.205404 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1f3a6b-dockercfg-r67lf\"" Nov 01 21:53:52.206702 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.206677 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 21:53:52.229626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.229604 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1f3a6b\"" Nov 01 21:53:52.298677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.298588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:52.377902 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.377863 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.378082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.377917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwtd\" (UniqueName: \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.479268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.479190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.479268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.479234 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwtd\" (UniqueName: \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.479900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.479879 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.492331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.492308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwtd\" (UniqueName: \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\") pod \"payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.511285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.511262 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:52.792082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.792049 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 21:53:52.794039 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:52.793995 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1833aa_234b_4245_b9e6_66a4938f0e87.slice/crio-529b35450d03d097116880bf824fcdd79c9386b0ad4b04de5437aab456d770f6 WatchSource:0}: Error finding container 529b35450d03d097116880bf824fcdd79c9386b0ad4b04de5437aab456d770f6: Status 404 returned error can't find the container with id 529b35450d03d097116880bf824fcdd79c9386b0ad4b04de5437aab456d770f6 Nov 01 21:53:52.825598 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.825563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:52.863094 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.863067 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 21:53:52.868236 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.868219 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:52.875239 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.875213 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 21:53:52.875557 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.875522 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-fb9597-dockercfg-g6k6j\"" Nov 01 21:53:52.955662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.955639 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb9597\"" Nov 01 21:53:52.958547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.958510 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:52.992225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.987051 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls7ff\" (UniqueName: \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:52.992225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:52.987305 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.088813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.088729 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7ff\" (UniqueName: \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.088813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.088805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.089508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.089489 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.100955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.100930 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls7ff\" (UniqueName: \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\") pod \"payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.183858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.183827 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:53.461427 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.461271 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:53.463547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.463515 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 21:53:53.585881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.585837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:53.592967 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.592931 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" event={"ID":"cf1833aa-234b-4245-b9e6-66a4938f0e87","Type":"ContainerStarted","Data":"529b35450d03d097116880bf824fcdd79c9386b0ad4b04de5437aab456d770f6"} Nov 01 21:53:53.596212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.596173 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" event={"ID":"dbfc35e8-62a7-46e5-9d9d-dedd613e0444","Type":"ContainerStarted","Data":"36963e1d4cb58e636a96aa6c523e5ccfaffdce02e06ea0442f5a3907c3f59632"} Nov 01 21:53:53.756724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.756628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:53.895309 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:53.895265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.174021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.173985 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.185373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.185336 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 21:53:54.193226 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.193198 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.198046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.198021 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 21:53:54.202693 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.202673 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-53b784\"" Nov 01 21:53:54.210149 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.210122 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 21:53:54.221284 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.221258 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.226807 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.226773 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 21:53:54.229195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.229174 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-53b784-dockercfg-wxpzp\"" Nov 01 21:53:54.299584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.299548 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.299762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.299603 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llzd\" (UniqueName: \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.307637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.307602 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.400289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.400253 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hnf\" (UniqueName: \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.400502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.400314 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.400502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.400433 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.400502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.400480 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llzd\" (UniqueName: \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.403272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.401814 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.415478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.413871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llzd\" (UniqueName: \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\") pod \"payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.501670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.501328 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hnf\" (UniqueName: \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.501670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.501404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.502436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.502412 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.523535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.523508 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:54.524678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.524648 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hnf\" (UniqueName: \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\") pod \"payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.543510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.543486 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:54.584469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.584414 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.726423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.726385 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.893333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.890885 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:54.900686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.900556 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 21:53:54.906348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:54.906323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 21:53:55.496896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.496850 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:55.506003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.505973 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 21:53:55.512122 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.512102 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.520637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.520614 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ff3e70-dockercfg-4qxmz\"" Nov 01 21:53:55.520734 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.520698 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 21:53:55.571773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.571652 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ff3e70\"" Nov 01 21:53:55.650472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.638174 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.650472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.638230 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdlj\" (UniqueName: \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.650472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.641122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:55.660425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.660384 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" event={"ID":"32ba525e-35ac-421c-9f61-4de3d469c3a0","Type":"ContainerStarted","Data":"8f3c6cdefeaef82a9e43c0597d18f0c5c7adf122e2aa051642c04f060782ddb4"} Nov 01 21:53:55.667164 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.667128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" event={"ID":"cf1833aa-234b-4245-b9e6-66a4938f0e87","Type":"ContainerStarted","Data":"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be"} Nov 01 21:53:55.668078 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.668057 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:53:55.669631 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.669603 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" event={"ID":"552811b9-12a3-45e2-a681-9640706a1fbf","Type":"ContainerStarted","Data":"7097ede72a9d0daf2246655ccb89cd50bd513cd0c7ef4ebf1a5e1ad9efa6e119"} Nov 01 21:53:55.687624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.687576 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" podStartSLOduration=1.029752722 podStartE2EDuration="3.68756308s" totalImagesPullingTime="2.657810358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:55.686647558 +0000 UTC m=+589.379628049" watchObservedRunningTime="2026-11-01 21:53:55.68756308 +0000 UTC m=+589.380543566" Nov 01 21:53:55.738765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.738737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.738883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.738781 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdlj\" (UniqueName: \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.739745 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.739722 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.753836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.753807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdlj\" (UniqueName: \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\") pod \"payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.846297 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.846265 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:55.876962 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:55.876905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.002104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.002016 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.133994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.133968 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 21:53:56.135697 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:56.135671 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656123f1_6928_46c7_9f4c_c668afb99662.slice/crio-536161e467e2f5a427e2c0533743368b3f7c95b209288ed1c1e6d309fb5e5b1c WatchSource:0}: Error finding container 536161e467e2f5a427e2c0533743368b3f7c95b209288ed1c1e6d309fb5e5b1c: Status 404 returned error can't find the container with id 536161e467e2f5a427e2c0533743368b3f7c95b209288ed1c1e6d309fb5e5b1c Nov 01 21:53:56.173179 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.173147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.390232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.390198 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.421916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.421883 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 21:53:56.426432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.426413 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.434338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.432263 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b0b7a3\"" Nov 01 21:53:56.434338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.433992 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b0b7a3-dockercfg-8fxlf\"" Nov 01 21:53:56.442000 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.441975 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 21:53:56.445316 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.445291 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2ql\" (UniqueName: \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.445402 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.445362 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.462679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.462648 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 21:53:56.466267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.466250 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.483212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.483187 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 21:53:56.515630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.515560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.556046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.556007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.556355 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.556321 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz54z\" (UniqueName: \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.556527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.556388 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.556602 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.556547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2ql\" (UniqueName: \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.557169 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.557144 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.569690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.569663 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2ql\" (UniqueName: \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\") pod \"payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.611689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.611660 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:53:56.657720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.657688 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz54z\" (UniqueName: \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.657871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.657739 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.658568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.658546 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.673774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.673745 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz54z\" (UniqueName: \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\") pod \"payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.682311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.682285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" event={"ID":"656123f1-6928-46c7-9f4c-c668afb99662","Type":"ContainerStarted","Data":"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1"} Nov 01 21:53:56.682519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.682485 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" event={"ID":"656123f1-6928-46c7-9f4c-c668afb99662","Type":"ContainerStarted","Data":"536161e467e2f5a427e2c0533743368b3f7c95b209288ed1c1e6d309fb5e5b1c"} Nov 01 21:53:56.682683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.682667 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:53:56.700083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.700050 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" event={"ID":"552811b9-12a3-45e2-a681-9640706a1fbf","Type":"ContainerStarted","Data":"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14"} Nov 01 21:53:56.700538 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.700515 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:53:56.702725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.702702 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" event={"ID":"dbfc35e8-62a7-46e5-9d9d-dedd613e0444","Type":"ContainerStarted","Data":"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c"} Nov 01 21:53:56.703028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.703013 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:53:56.708613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.708577 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" event={"ID":"32ba525e-35ac-421c-9f61-4de3d469c3a0","Type":"ContainerStarted","Data":"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629"} Nov 01 21:53:56.708766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.708753 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:53:56.710400 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.710348 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" podStartSLOduration=1.710331243 podStartE2EDuration="1.710331243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:56.704355009 +0000 UTC m=+590.397335497" watchObservedRunningTime="2026-11-01 21:53:56.710331243 +0000 UTC m=+590.403311730" Nov 01 21:53:56.732436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.732386 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" podStartSLOduration=1.9351879749999998 podStartE2EDuration="4.73236927s" totalImagesPullingTime="2.797181295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:56.729318347 +0000 UTC m=+590.422298836" watchObservedRunningTime="2026-11-01 21:53:56.73236927 +0000 UTC m=+590.425349754" Nov 01 21:53:56.744132 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.744103 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:56.753543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.753490 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" podStartSLOduration=1.938926734 podStartE2EDuration="2.753471932s" totalImagesPullingTime="814.545198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:56.752915578 +0000 UTC m=+590.445896081" watchObservedRunningTime="2026-11-01 21:53:56.753471932 +0000 UTC m=+590.446452419" Nov 01 21:53:56.769357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.769328 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:56.780122 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.780073 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" podStartSLOduration=1.998125415 podStartE2EDuration="2.780059051s" totalImagesPullingTime="781.933636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:56.776253995 +0000 UTC m=+590.469234483" watchObservedRunningTime="2026-11-01 21:53:56.780059051 +0000 UTC m=+590.473039538" Nov 01 21:53:56.780552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.780529 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:56.893954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:56.893918 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:57.037519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.037493 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 21:53:57.039644 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:57.039609 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5913db_ddc3_4547_bb1d_c4a908c318e7.slice/crio-4cae792eb3d4531d997cd25a60500eedc25be9b552da6adcf7736efd11826864 WatchSource:0}: Error finding container 4cae792eb3d4531d997cd25a60500eedc25be9b552da6adcf7736efd11826864: Status 404 returned error can't find the container with id 4cae792eb3d4531d997cd25a60500eedc25be9b552da6adcf7736efd11826864 Nov 01 21:53:57.041006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.040942 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:57.075324 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.075299 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 21:53:57.077651 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:53:57.077624 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ad7920_1e49_4002_9ca8_f717e36114bb.slice/crio-66b36ccc60defd505dbd04d31488a9de67f2adad16a9f4fa7907747d817c1759 WatchSource:0}: Error finding container 66b36ccc60defd505dbd04d31488a9de67f2adad16a9f4fa7907747d817c1759: Status 404 returned error can't find the container with id 66b36ccc60defd505dbd04d31488a9de67f2adad16a9f4fa7907747d817c1759 Nov 01 21:53:57.716408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.715881 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" event={"ID":"cb5913db-ddc3-4547-bb1d-c4a908c318e7","Type":"ContainerStarted","Data":"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f"} Nov 01 21:53:57.716883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.716858 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:53:57.717021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.716895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" event={"ID":"cb5913db-ddc3-4547-bb1d-c4a908c318e7","Type":"ContainerStarted","Data":"4cae792eb3d4531d997cd25a60500eedc25be9b552da6adcf7736efd11826864"} Nov 01 21:53:57.718301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.718272 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" event={"ID":"a6ad7920-1e49-4002-9ca8-f717e36114bb","Type":"ContainerStarted","Data":"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca"} Nov 01 21:53:57.718424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.718328 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" event={"ID":"a6ad7920-1e49-4002-9ca8-f717e36114bb","Type":"ContainerStarted","Data":"66b36ccc60defd505dbd04d31488a9de67f2adad16a9f4fa7907747d817c1759"} Nov 01 21:53:57.718424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.718359 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:53:57.721248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.721215 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:57.737351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.737311 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" podStartSLOduration=1.737299001 podStartE2EDuration="1.737299001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:57.733704412 +0000 UTC m=+591.426684900" watchObservedRunningTime="2026-11-01 21:53:57.737299001 +0000 UTC m=+591.430279524" Nov 01 21:53:57.752682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.752641 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" podStartSLOduration=1.752627755 podStartE2EDuration="1.752627755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:57.75130599 +0000 UTC m=+591.444286476" watchObservedRunningTime="2026-11-01 21:53:57.752627755 +0000 UTC m=+591.445608241" Nov 01 21:53:57.842397 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:57.842353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:58.560541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:58.560502 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:59.395974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:59.395935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:53:59.522648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:53:59.522611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:00.151044 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:00.151004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:00.285957 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:00.285926 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:00.290048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:00.290014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:00.475344 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:00.475261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:01.149659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:01.149627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:01.281910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:01.281867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:02.034038 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:02.034001 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:02.180317 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:02.180274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:03.110810 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:03.110774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:03.963490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:03.963439 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:04.346907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:04.346873 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:05.223418 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:05.223378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:06.332764 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:06.332727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:06.512355 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:06.512327 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:54:06.512692 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:06.512675 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:54:07.409726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:07.409698 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:07.721379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:07.721296 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 21:54:07.721379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:07.721374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 21:54:07.721621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:07.721409 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 21:54:07.721621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:07.721584 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 21:54:08.705614 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:08.705578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:08.727227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:08.727200 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 21:54:08.727581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:08.727558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 21:54:08.728108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:08.728085 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 21:54:09.618550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:09.618517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:10.609826 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:10.609788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:10.739488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:10.739428 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.190821 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.190793 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.319624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.319581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.454893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.454807 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.581916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.581883 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.728638 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.728544 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.869319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:11.869282 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.048054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.048017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.207876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.207842 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 21:54:12.217147 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.217119 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.219815 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.219789 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 21:54:12.226871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.226718 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"default-dockercfg-bqm9h\"" Nov 01 21:54:12.227620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.227320 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"simulator-w0-1f3a6b-kserve-self-signed-certs\"" Nov 01 21:54:12.232109 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.232091 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"openshift-service-ca.crt\"" Nov 01 21:54:12.232206 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.232114 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"kube-root-ca.crt\"" Nov 01 21:54:12.272366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.272333 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.316552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.316468 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.316766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.316737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7sgb\" (UniqueName: \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.316920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.316894 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.317017 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.316985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.317176 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.317154 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.317242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.317197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418559 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418584 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7sgb\" (UniqueName: \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418692 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.418956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.419016 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.418992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.419069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.419052 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.420920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.420896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.421408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.421386 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.431848 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.431818 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7sgb\" (UniqueName: \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\") pod \"simulator-w0-1f3a6b-kserve-654b54d558-m64vf\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.480244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.480201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.538935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.538906 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:12.622738 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.622517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.831936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.831912 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 21:54:12.833813 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:12.833784 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2861c4e5_27aa_479a_9fc4_e0431c665724.slice/crio-d91302013687404755d6d156447eb0e4431cf591111db0e11ca6f014f21939f4 WatchSource:0}: Error finding container d91302013687404755d6d156447eb0e4431cf591111db0e11ca6f014f21939f4: Status 404 returned error can't find the container with id d91302013687404755d6d156447eb0e4431cf591111db0e11ca6f014f21939f4 Nov 01 21:54:12.943672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:12.943594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:13.826705 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:13.826672 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" event={"ID":"2861c4e5-27aa-479a-9fc4-e0431c665724","Type":"ContainerStarted","Data":"d91302013687404755d6d156447eb0e4431cf591111db0e11ca6f014f21939f4"} Nov 01 21:54:14.020124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:14.020079 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:14.834075 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:14.834019 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" event={"ID":"2861c4e5-27aa-479a-9fc4-e0431c665724","Type":"ContainerStarted","Data":"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2"} Nov 01 21:54:14.855695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:14.855634 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podStartSLOduration=1.5957343750000001 podStartE2EDuration="2.855616525s" totalImagesPullingTime="1.25988215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:14.855184611 +0000 UTC m=+608.548165097" watchObservedRunningTime="2026-11-01 21:54:14.855616525 +0000 UTC m=+608.548597011" Nov 01 21:54:14.860968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:14.860946 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:15.235159 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:15.235077 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:15.539291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:15.539259 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:15.826722 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:15.826634 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.172311 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.172216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.304675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.304636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.445151 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.445069 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.585584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.585548 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.751398 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.751300 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:16.891302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:16.891264 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:17.019855 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.019820 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:17.153157 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.152842 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 21:54:17.166790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.166756 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 21:54:17.166951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.166849 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.180621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.178266 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"kube-root-ca.crt\"" Nov 01 21:54:17.180621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.178587 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"openshift-service-ca.crt\"" Nov 01 21:54:17.180621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.178697 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"default-dockercfg-kwc4v\"" Nov 01 21:54:17.180621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.178782 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"simulator-w4-0def36-kserve-self-signed-certs\"" Nov 01 21:54:17.226270 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.226231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:17.287145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287072 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.287145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59ks\" (UniqueName: \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.287308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.287355 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.287410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287393 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.287519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.287504 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388172 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388228 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388284 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59ks\" (UniqueName: \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388401 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388649 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.388717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388671 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.389041 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.388904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.390793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.390765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.391044 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.391018 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.438798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.438773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59ks\" (UniqueName: \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\") pod \"simulator-w4-0def36-kserve-57d778fb89-cpdqw\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.489622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.489597 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:17.774607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.774579 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 21:54:17.776711 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:17.776680 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a5df55_6d1d_4d2b_863d_e9c939863d6a.slice/crio-9b78b335e7cd386fb3653bc40695160abd7d3d9ea9103be232082028f02160bd WatchSource:0}: Error finding container 9b78b335e7cd386fb3653bc40695160abd7d3d9ea9103be232082028f02160bd: Status 404 returned error can't find the container with id 9b78b335e7cd386fb3653bc40695160abd7d3d9ea9103be232082028f02160bd Nov 01 21:54:17.856324 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.856293 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" event={"ID":"20a5df55-6d1d-4d2b-863d-e9c939863d6a","Type":"ContainerStarted","Data":"2a45fd11e8a959c4208f13142a4366d4220e4170c0cb658bf87d0922d7824c22"} Nov 01 21:54:17.856494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.856341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" event={"ID":"20a5df55-6d1d-4d2b-863d-e9c939863d6a","Type":"ContainerStarted","Data":"9b78b335e7cd386fb3653bc40695160abd7d3d9ea9103be232082028f02160bd"} Nov 01 21:54:17.878541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.878386 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podStartSLOduration=0.878366869 podStartE2EDuration="878.366869ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:17.877526028 +0000 UTC m=+611.570506515" watchObservedRunningTime="2026-11-01 21:54:17.878366869 +0000 UTC m=+611.571347355" Nov 01 21:54:17.916329 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:17.916298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:18.490087 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:18.490048 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:18.821278 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:18.821241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:18.862244 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:54:18.862203 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:54:19.310991 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.310951 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 21:54:19.315842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.315814 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.327119 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.327092 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"kube-root-ca.crt\"" Nov 01 21:54:19.327294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.327156 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"openshift-service-ca.crt\"" Nov 01 21:54:19.327294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.327197 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"simulator-w3-ff3e70-kserve-self-signed-certs\"" Nov 01 21:54:19.330666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.330643 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"default-dockercfg-prhv9\"" Nov 01 21:54:19.331410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.331386 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 21:54:19.395588 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.395554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:19.412877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.412844 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.413032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.412971 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.413032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.412999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.413121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.413069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.413207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.413187 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xxb\" (UniqueName: \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.413331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.413308 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.514566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.514529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.515176 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.515259 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.515289 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.515318 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.515420 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xxb\" (UniqueName: \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.515778 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.514969 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.516737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.516710 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.516975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.516956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.518743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.518715 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.519125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.519106 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.528686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.528650 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xxb\" (UniqueName: \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\") pod \"simulator-w3-ff3e70-kserve-84cf44d4c-zckj9\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.632482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.632374 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:19.713813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.713785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:19.923637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:19.923594 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 21:54:19.925321 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:19.925282 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda336141c_c982_40b9_a358_6cc095f59f38.slice/crio-610c8171ff465a605a58d2bedc451cc703789431fe83955286a254269a3361b3 WatchSource:0}: Error finding container 610c8171ff465a605a58d2bedc451cc703789431fe83955286a254269a3361b3: Status 404 returned error can't find the container with id 610c8171ff465a605a58d2bedc451cc703789431fe83955286a254269a3361b3 Nov 01 21:54:20.101864 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:20.101830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:20.655552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:20.655517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:20.876255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:20.876217 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" event={"ID":"a336141c-c982-40b9-a358-6cc095f59f38","Type":"ContainerStarted","Data":"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54"} Nov 01 21:54:20.876438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:20.876265 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" event={"ID":"a336141c-c982-40b9-a358-6cc095f59f38","Type":"ContainerStarted","Data":"610c8171ff465a605a58d2bedc451cc703789431fe83955286a254269a3361b3"} Nov 01 21:54:20.898631 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:20.898572 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podStartSLOduration=1.898554911 podStartE2EDuration="1.898554911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:20.895849374 +0000 UTC m=+614.588829862" watchObservedRunningTime="2026-11-01 21:54:20.898554911 +0000 UTC m=+614.591535405" Nov 01 21:54:21.436688 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:21.436652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:21.632462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:21.632422 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:22.309597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.309559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:22.540127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.540092 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:22.553381 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.553351 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:22.910707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.910680 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:54:22.979839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.976106 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 21:54:22.987723 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.987692 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 21:54:22.987883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.987802 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:22.992633 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.992604 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"default-dockercfg-77f4g\"" Nov 01 21:54:22.992894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.992874 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"openshift-service-ca.crt\"" Nov 01 21:54:22.993048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:22.993029 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"kube-root-ca.crt\"" Nov 01 21:54:23.003378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.003354 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"simulator-w6-fb9597-kserve-self-signed-certs\"" Nov 01 21:54:23.033671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.033641 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:23.157904 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.157867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:23.169655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169565 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.169655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.169883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.169883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169793 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.169883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169840 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.170042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.169920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfhx\" (UniqueName: \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfhx\" (UniqueName: \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271193 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271260 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271401 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271790 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.271949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271928 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.272017 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.271947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.273780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.273751 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.273893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.273876 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.285980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.285959 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfhx\" (UniqueName: \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\") pod \"simulator-w6-fb9597-kserve-958d57475-45f4h\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.309127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.309105 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:23.594900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.594864 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 21:54:23.596431 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:23.596409 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813df7b3_d181_4675_a695_af946f32a232.slice/crio-d623482b8e1e210a5aa1a16c4f3f3edcc6bc6c305834638575d3af16fede3d63 WatchSource:0}: Error finding container d623482b8e1e210a5aa1a16c4f3f3edcc6bc6c305834638575d3af16fede3d63: Status 404 returned error can't find the container with id d623482b8e1e210a5aa1a16c4f3f3edcc6bc6c305834638575d3af16fede3d63 Nov 01 21:54:23.897206 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.897091 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:23.907737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.907701 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" event={"ID":"813df7b3-d181-4675-a695-af946f32a232","Type":"ContainerStarted","Data":"164ea1cd00c9424db37d3765403d9e9b1cf2661e4f669a352c812472187e6e53"} Nov 01 21:54:23.907896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.907755 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" event={"ID":"813df7b3-d181-4675-a695-af946f32a232","Type":"ContainerStarted","Data":"d623482b8e1e210a5aa1a16c4f3f3edcc6bc6c305834638575d3af16fede3d63"} Nov 01 21:54:23.929837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:23.929779 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podStartSLOduration=1.929762352 podStartE2EDuration="1.929762352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:23.927340265 +0000 UTC m=+617.620320752" watchObservedRunningTime="2026-11-01 21:54:23.929762352 +0000 UTC m=+617.622742841" Nov 01 21:54:24.309955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:24.309919 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:24.719022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:24.718938 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:25.447517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:25.447477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:25.584496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:25.580780 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:25.890389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:25.890346 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:26.351714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:26.351675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:27.101198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:27.101153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:27.490697 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:27.490610 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:27.504170 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:27.504141 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:27.780973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:27.780935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:27.944499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:27.944469 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:54:28.463045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:28.463012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:28.588780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:28.588738 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:29.239968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:29.239913 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:29.633348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:29.633306 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:29.650703 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:29.650675 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:29.947947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:29.947868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:29.958953 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:29.958924 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:54:30.647693 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:30.647664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:31.425953 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:31.425910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:32.137054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:32.137017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:32.956349 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:32.956309 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:33.309874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:33.309836 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:33.324121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:33.324095 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:33.894154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:33.894116 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:33.987586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:33.987558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:54:34.732354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:34.732325 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:35.446471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:35.446412 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:36.139358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:36.139319 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:36.837804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:36.837765 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:37.407034 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:37.407002 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:37.539913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:37.539875 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:38.230134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:38.230096 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:39.036136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:39.036104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:39.704940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:39.704910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:40.292378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:40.292340 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:40.424762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:40.424727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:41.129532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:41.129489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:41.956523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:41.956484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:42.652824 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:42.652785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:43.373568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:43.373527 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:43.980342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:43.980305 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 21:54:43.986478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:43.986434 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:43.992971 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:43.992947 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"premium-w4-0def36-kserve-self-signed-certs\"" Nov 01 21:54:44.008212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.008185 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 21:54:44.069656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.069621 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:44.099417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78cf\" (UniqueName: \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.099577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099455 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.099624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099591 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.099677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099660 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.099788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099765 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.099839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.099823 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201052 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201114 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78cf\" (UniqueName: \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201269 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201334 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201531 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201692 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.201960 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.201931 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.204150 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.204122 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.205029 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.205005 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.213630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.213608 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78cf\" (UniqueName: \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\") pod \"premium-w4-0def36-kserve-8db555bdd-z2jg8\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.292003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.291970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:44.302517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.302487 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:44.590273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.590238 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 21:54:44.591227 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:44.591188 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fd260ba_f2e1_4fa0_a46f_925269a37195.slice/crio-3f9f5b1e9622ec57605fd1879f43a6c3d08fa19fc815a851f0367c23b4c8893f WatchSource:0}: Error finding container 3f9f5b1e9622ec57605fd1879f43a6c3d08fa19fc815a851f0367c23b4c8893f: Status 404 returned error can't find the container with id 3f9f5b1e9622ec57605fd1879f43a6c3d08fa19fc815a851f0367c23b4c8893f Nov 01 21:54:44.949769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:44.949679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:45.054869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.054831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" event={"ID":"2fd260ba-f2e1-4fa0-a46f-925269a37195","Type":"ContainerStarted","Data":"4af9442e9b9bba641411431c629c827ba8c5315c988933bacc03919886ad693d"} Nov 01 21:54:45.055054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.054881 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" event={"ID":"2fd260ba-f2e1-4fa0-a46f-925269a37195","Type":"ContainerStarted","Data":"3f9f5b1e9622ec57605fd1879f43a6c3d08fa19fc815a851f0367c23b4c8893f"} Nov 01 21:54:45.075090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.075046 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podStartSLOduration=2.075032132 podStartE2EDuration="2.075032132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:45.073298184 +0000 UTC m=+638.766278671" watchObservedRunningTime="2026-11-01 21:54:45.075032132 +0000 UTC m=+638.768012664" Nov 01 21:54:45.178718 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.178674 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:45.302986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.302951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:45.743964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:45.743867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:46.450915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.450884 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:46.713572 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.713492 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 21:54:46.718847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.718818 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.728920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.728897 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"openshift-service-ca.crt\"" Nov 01 21:54:46.729752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.729728 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"default-dockercfg-rr98w\"" Nov 01 21:54:46.730480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.730422 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"premium-w5-b0b7a3-kserve-self-signed-certs\"" Nov 01 21:54:46.730597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.730557 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"kube-root-ca.crt\"" Nov 01 21:54:46.735542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.735519 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 21:54:46.790184 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.790147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:46.829295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829261 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.829510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829332 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.829510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.829638 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829565 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.829638 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829608 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.829745 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.829641 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477q7\" (UniqueName: \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.930306 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930266 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.930504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930388 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.930504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930425 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.930630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930533 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.930890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930857 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.931006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.931006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930939 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.931006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.930999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477q7\" (UniqueName: \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.931265 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.931241 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.932776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.932747 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.932857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.932825 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:46.946071 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:46.946046 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-477q7\" (UniqueName: \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\") pod \"premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:47.037046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:47.037014 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:47.152242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:47.152207 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:47.323368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:47.323333 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 21:54:47.325137 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:54:47.325104 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aabc696_b098_4ffa_b640_7fbf8eddf008.slice/crio-6416a44f47adc72f071175a629dd6fadab2f63817ddfe0f97b794c567bea9938 WatchSource:0}: Error finding container 6416a44f47adc72f071175a629dd6fadab2f63817ddfe0f97b794c567bea9938: Status 404 returned error can't find the container with id 6416a44f47adc72f071175a629dd6fadab2f63817ddfe0f97b794c567bea9938 Nov 01 21:54:47.944950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:47.944914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:48.077102 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:48.077064 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" event={"ID":"9aabc696-b098-4ffa-b640-7fbf8eddf008","Type":"ContainerStarted","Data":"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89"} Nov 01 21:54:48.077279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:48.077114 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" event={"ID":"9aabc696-b098-4ffa-b640-7fbf8eddf008","Type":"ContainerStarted","Data":"6416a44f47adc72f071175a629dd6fadab2f63817ddfe0f97b794c567bea9938"} Nov 01 21:54:48.102184 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:48.102137 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podStartSLOduration=2.1021227319999998 podStartE2EDuration="2.102122732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:48.09914607 +0000 UTC m=+641.792126558" watchObservedRunningTime="2026-11-01 21:54:48.102122732 +0000 UTC m=+641.795103218" Nov 01 21:54:48.646351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:48.646316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:48.862861 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:54:48.862817 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:54:49.037171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:49.037130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:49.445985 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:49.445894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:50.143846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:50.143814 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:50.842915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:50.842873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:51.459720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:51.459678 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:54:51.554697 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:51.554649 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:52.445473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:52.445404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:52.604940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:52.604894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:53.341725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:53.341678 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:54.128672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:54.128633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:54.303231 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:54.303199 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:54.316046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:54.316020 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:54.780362 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:54.780319 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:55.135429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:55.135361 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:54:55.545789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:55.545754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:56.220989 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:56.220951 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:56.345876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:56.345839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:57.003883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:57.003835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:57.038061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:57.038027 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:57.050936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:57.050912 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:57.150047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:57.150020 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:54:57.713039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:57.713004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:58.533678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:58.533631 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:58.818974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:58.818880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:59.335120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:54:59.335083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:00.036555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.036521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:00.343618 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.343535 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 21:55:00.348279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.348258 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.357753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.357727 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"kube-root-ca.crt\"" Nov 01 21:55:00.358554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.358514 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"openshift-service-ca.crt\"" Nov 01 21:55:00.358714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.358692 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"default-dockercfg-pc87z\"" Nov 01 21:55:00.358798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.358729 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"premium-w2-f033b1-kserve-self-signed-certs\"" Nov 01 21:55:00.363120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.363098 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 21:55:00.371685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371653 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.371800 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371777 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.371872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371854 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj74h\" (UniqueName: \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.371930 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.371985 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371952 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.372044 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.371986 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.415929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.415891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:00.472574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.472756 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.472858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj74h\" (UniqueName: \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.472923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472877 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.472979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472947 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.473039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.472999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.473097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.473063 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.473626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.473599 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.473847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.473827 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.475485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.475429 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.475822 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.475801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.489470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.486188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj74h\" (UniqueName: \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\") pod \"premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.664074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.664004 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:00.905592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.905554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:00.949398 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:00.949369 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 21:55:00.951214 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:00.951185 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb679209_52cd_41f0_bde3_9d6a6549d4ad.slice/crio-013e749f32f4f11381203f1de42891fb6c931c70c45dc8145fe51c2b73a2d201 WatchSource:0}: Error finding container 013e749f32f4f11381203f1de42891fb6c931c70c45dc8145fe51c2b73a2d201: Status 404 returned error can't find the container with id 013e749f32f4f11381203f1de42891fb6c931c70c45dc8145fe51c2b73a2d201 Nov 01 21:55:01.166784 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:01.166749 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" event={"ID":"eb679209-52cd-41f0-bde3-9d6a6549d4ad","Type":"ContainerStarted","Data":"5f329d9ba1bc4c50ec62ae6ba4f4c4e499cd4fd6f9ee1799634df4416290d13c"} Nov 01 21:55:01.167195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:01.166792 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" event={"ID":"eb679209-52cd-41f0-bde3-9d6a6549d4ad","Type":"ContainerStarted","Data":"013e749f32f4f11381203f1de42891fb6c931c70c45dc8145fe51c2b73a2d201"} Nov 01 21:55:01.189130 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:01.189067 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podStartSLOduration=1.189048826 podStartE2EDuration="1.189048826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:01.186549622 +0000 UTC m=+654.879530108" watchObservedRunningTime="2026-11-01 21:55:01.189048826 +0000 UTC m=+654.882029314" Nov 01 21:55:01.664181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:01.664147 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:01.733409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:01.733374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:02.442088 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:02.442056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:03.142385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:03.142353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:03.832573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:03.832534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:04.649924 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:04.649880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:05.497705 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:05.497668 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:06.499060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:06.499023 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:07.232259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:07.232219 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:07.659787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:07.659754 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:55:08.050757 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:08.050712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:08.940352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:08.940305 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:09.831057 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:09.831018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:10.641079 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:10.641036 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:10.664357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:10.664320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:10.677528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:10.677503 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:11.247998 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:11.247973 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:55:11.313998 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:11.313964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:12.024653 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:12.024613 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:12.836108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:12.836072 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:13.642262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:13.642169 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:14.331293 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:14.331255 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:14.998009 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:14.997968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:15.941013 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:15.940965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:16.861737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:16.861703 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:17.788856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.788771 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:17.883378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.883346 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 21:55:17.888548 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.888532 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.896713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.896694 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b0b7a3-openshift-default-dockercfg-rxv7b\"" Nov 01 21:55:17.900403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.900376 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 21:55:17.968407 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968379 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968417 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968534 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968637 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968787 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:17.968933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:17.968901 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069663 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069663 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069624 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069759 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069814 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.069867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069915 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.069986 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.070051 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.070313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.070345 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.070403 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.070524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.070426 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.072154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.072125 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.072326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.072308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.082086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.082061 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.082213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.082198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.202436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.202408 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:18.486827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.486800 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 21:55:18.487988 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:18.487957 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bae2a6_5630_4751_82bc_6ae358c9e943.slice/crio-81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf WatchSource:0}: Error finding container 81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf: Status 404 returned error can't find the container with id 81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf Nov 01 21:55:18.490384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.490351 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:18.490504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.490417 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:18.490504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.490481 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:18.636511 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:18.636474 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:18.861936 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:55:18.861894 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:55:19.307468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:19.306229 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" event={"ID":"03bae2a6-5630-4751-82bc-6ae358c9e943","Type":"ContainerStarted","Data":"7c1e2d899f42ed385097e16fec88287eea8c5aa00192ad3c6bb291e665796fc0"} Nov 01 21:55:19.307468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:19.306297 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" event={"ID":"03bae2a6-5630-4751-82bc-6ae358c9e943","Type":"ContainerStarted","Data":"81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf"} Nov 01 21:55:19.335070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:19.335005 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podStartSLOduration=2.334985038 podStartE2EDuration="2.334985038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:19.332622668 +0000 UTC m=+673.025603155" watchObservedRunningTime="2026-11-01 21:55:19.334985038 +0000 UTC m=+673.027965524" Nov 01 21:55:19.626269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:19.626167 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:20.208716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:20.208675 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:20.210249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:20.210196 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 21:55:20.210399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:20.210274 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 21:55:20.458858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:20.458775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:21.205474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:21.204821 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 21:55:21.205474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:21.204880 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 21:55:21.258353 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:21.258324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:22.063391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:22.063353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:22.207891 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:22.207852 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 21:55:22.208095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:22.208014 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 21:55:22.942525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:22.942484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:23.202981 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:23.202896 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 21:55:23.203583 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:23.203547 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 21:55:23.650898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:23.650849 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:24.203878 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:24.203838 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 01 21:55:24.204389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:24.203905 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 01 21:55:24.425888 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:24.425842 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:25.125817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:25.125782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:25.208254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:25.208221 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:25.208672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:25.208502 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:25.209257 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:25.209243 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 21:55:25.917636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:25.917601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:26.628025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:26.627992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:27.322176 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:27.322137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:28.007854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:28.007824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:28.708614 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:28.708582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:29.434990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:29.434952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:30.139871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:30.139823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:30.812492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:30.812462 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:55:30.830833 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:30.830798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:31.825057 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:31.825017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:32.643021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:32.642972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:32.981567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:32.981484 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 21:55:32.989371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:32.989337 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.003515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:32.996742 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-53b784-openshift-default-dockercfg-m5k4p\"" Nov 01 21:55:33.004976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.004948 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 21:55:33.046747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.046711 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.046896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.046764 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.046950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.046914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047001 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.046950 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nqx\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047001 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.046991 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.047046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.047216 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.047256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.047308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.047282 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.148537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148503 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.148776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.148913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.148913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148820 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.148913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148847 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148927 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.148961 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nqx\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149003 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149039 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149173 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149208 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149425 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.149857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.149837 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.150073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.150043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.151590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.151565 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.153193 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.153169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.161310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.161289 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.161847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.161827 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nqx\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.320839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.320806 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:33.538457 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.538400 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:33.614561 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.614521 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 21:55:33.614721 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:33.614695 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a0abae_96d9_4d9a_beaa_97e6d61922e8.slice/crio-39721ce3db145a59f2d841ccbf2abdfcb1d9975ed3636aab3531876a15769604 WatchSource:0}: Error finding container 39721ce3db145a59f2d841ccbf2abdfcb1d9975ed3636aab3531876a15769604: Status 404 returned error can't find the container with id 39721ce3db145a59f2d841ccbf2abdfcb1d9975ed3636aab3531876a15769604 Nov 01 21:55:33.617248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.617214 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:33.617350 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.617277 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:33.617350 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:33.617307 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:34.321967 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.321930 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2"] Nov 01 21:55:34.325885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.325866 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.331403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.331381 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:55:34.332778 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.332745 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:55:34.332889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.332810 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:55:34.336500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.333360 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:55:34.337153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.337134 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2"] Nov 01 21:55:34.355976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.355952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:34.363010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.362988 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lc8\" (UniqueName: \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.363216 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.363192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.363295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.363262 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.421727 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.421692 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" event={"ID":"90a0abae-96d9-4d9a-beaa-97e6d61922e8","Type":"ContainerStarted","Data":"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1"} Nov 01 21:55:34.421925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.421755 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" event={"ID":"90a0abae-96d9-4d9a-beaa-97e6d61922e8","Type":"ContainerStarted","Data":"39721ce3db145a59f2d841ccbf2abdfcb1d9975ed3636aab3531876a15769604"} Nov 01 21:55:34.448628 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.448572 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podStartSLOduration=2.448556505 podStartE2EDuration="2.448556505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:34.44446641 +0000 UTC m=+688.137446905" watchObservedRunningTime="2026-11-01 21:55:34.448556505 +0000 UTC m=+688.141536990" Nov 01 21:55:34.464264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.464210 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.464407 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.464375 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.464550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.464517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lc8\" (UniqueName: \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.465320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.465290 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.466809 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.466792 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.495470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.476873 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lc8\" (UniqueName: \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\") pod \"maas-api-revoke-sub-90cfc7a4-2wzp2\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.640986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.640899 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:34.953433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.953407 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2"] Nov 01 21:55:34.955464 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:34.955421 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc59da_6939_4dd7_9a28_fd8a44ee92fa.slice/crio-22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97 WatchSource:0}: Error finding container 22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97: Status 404 returned error can't find the container with id 22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97 Nov 01 21:55:34.957586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:34.957565 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:55:35.038831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.038795 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:35.321902 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.321870 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:35.324056 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.323397 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:55:35.324056 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.323439 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 21:55:35.429111 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.429071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" event={"ID":"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa","Type":"ContainerStarted","Data":"22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97"} Nov 01 21:55:35.738672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:35.738581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:36.321946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:36.321910 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:55:36.322127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:36.321965 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 21:55:36.443867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:36.443825 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:37.213820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:37.213736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:37.322097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:37.322052 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:55:37.322293 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:37.322138 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 21:55:37.925523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:37.925477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:38.321717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:38.321674 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:55:38.321926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:38.321757 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 21:55:38.620672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:38.620012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:39.321805 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:39.321777 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:55:39.322117 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:39.321827 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 21:55:39.357361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:39.357330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:39.456844 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:39.456809 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" event={"ID":"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa","Type":"ContainerStarted","Data":"3fd41024baf72c958c6161072b6eb5511f130edd015d3862de90743d64abf8b2"} Nov 01 21:55:39.478517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:39.478420 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" podStartSLOduration=1.495843259 podStartE2EDuration="5.478400745s" totalImagesPullingTime="3.982557486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:39.475982688 +0000 UTC m=+693.168963175" watchObservedRunningTime="2026-11-01 21:55:39.478400745 +0000 UTC m=+693.171381232" Nov 01 21:55:40.037621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.037584 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:40.326167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.326137 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:40.326699 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.326563 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:40.327275 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.327254 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 21:55:40.465592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.465558 5036 generic.go:356] "Generic (PLEG): container finished" podID="f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" containerID="3fd41024baf72c958c6161072b6eb5511f130edd015d3862de90743d64abf8b2" exitCode=0 Nov 01 21:55:40.465785 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.465642 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" event={"ID":"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa","Type":"ContainerDied","Data":"3fd41024baf72c958c6161072b6eb5511f130edd015d3862de90743d64abf8b2"} Nov 01 21:55:40.742897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:40.742806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:41.631008 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.630981 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:41.639045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.639018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:41.765841 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.765763 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\" (UniqueName: \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\") pod \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " Nov 01 21:55:41.765987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.765905 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\") pod \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " Nov 01 21:55:41.766039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.766023 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\") pod \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\" (UID: \"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa\") " Nov 01 21:55:41.766403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.766378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca" pod "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" (UID: "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:41.767990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.767967 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca" pod "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" (UID: "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:41.768074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.768054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8" pod "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" (UID: "f8bc59da-6939-4dd7-9a28-fd8a44ee92fa"). InnerVolumeSpecName "kube-api-access-p5lc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:41.867817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.867784 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:41.867817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.867812 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:41.867817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:41.867821 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lc8\" (UniqueName: \"kubernetes.io/projected/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa-kube-api-access-p5lc8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:42.452644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:42.452615 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:42.479452 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:42.479421 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" event={"ID":"f8bc59da-6939-4dd7-9a28-fd8a44ee92fa","Type":"ContainerDied","Data":"22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97"} Nov 01 21:55:42.479583 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:42.479485 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ef3fb6778b91381a20b6610f45e35032b63dc0396bd8e1dfabc21d0f63fd97" Nov 01 21:55:42.479583 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:42.479431 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2" Nov 01 21:55:43.419904 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.419873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:43.652058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.652025 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p"] Nov 01 21:55:43.652680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.652659 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" containerName="revoke-keys" Nov 01 21:55:43.652680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.652681 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" containerName="revoke-keys" Nov 01 21:55:43.652883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.652865 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" containerName="revoke-keys" Nov 01 21:55:43.656673 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.656656 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.664834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.664809 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:55:43.668894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.668871 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p"] Nov 01 21:55:43.669131 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.669113 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:55:43.669216 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.669198 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:55:43.669279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.669237 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:55:43.786599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.786569 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.786799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.786610 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.786799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.786728 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcs9\" (UniqueName: \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.888210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.888174 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.888210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.888209 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.888462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.888267 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcs9\" (UniqueName: \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.889054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.889031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.890577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.890553 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.903460 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.900697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcs9\" (UniqueName: \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\") pod \"maas-api-revoke-sub-2339d468-fj97p\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:43.973583 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:43.973559 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:44.264903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.264874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p"] Nov 01 21:55:44.265881 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:44.265855 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8d920b_088b_4180_9d1b_91741afa7c7a.slice/crio-349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d WatchSource:0}: Error finding container 349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d: Status 404 returned error can't find the container with id 349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d Nov 01 21:55:44.315934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.315904 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8"] Nov 01 21:55:44.321267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.321248 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.327911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.327616 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8"] Nov 01 21:55:44.394915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.394890 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22w7d\" (UniqueName: \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.395036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.394958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.395077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.395031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.484750 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.484720 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:44.491698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.491670 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" event={"ID":"6d8d920b-088b-4180-9d1b-91741afa7c7a","Type":"ContainerStarted","Data":"349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d"} Nov 01 21:55:44.496572 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.496547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.496677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.496581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.496677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.496654 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22w7d\" (UniqueName: \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.497382 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.497361 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.499142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.499120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.510771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.510737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22w7d\" (UniqueName: \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\") pod \"maas-api-revoke-sub-54775354-4j6l8\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.640456 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.640414 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:44.922956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:44.922921 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8"] Nov 01 21:55:44.924472 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:44.924409 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9384ed2e_f8bf_4f7f_9ae2_dd868e325237.slice/crio-e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e WatchSource:0}: Error finding container e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e: Status 404 returned error can't find the container with id e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e Nov 01 21:55:45.235658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.235628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:45.500419 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.500335 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d8d920b-088b-4180-9d1b-91741afa7c7a" containerID="656cf6b64e42209c59cd26b0d0cf135e853432e2a3ce106304ae001221bc1c94" exitCode=0 Nov 01 21:55:45.500867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.500420 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" event={"ID":"6d8d920b-088b-4180-9d1b-91741afa7c7a","Type":"ContainerDied","Data":"656cf6b64e42209c59cd26b0d0cf135e853432e2a3ce106304ae001221bc1c94"} Nov 01 21:55:45.502263 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.502239 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" event={"ID":"9384ed2e-f8bf-4f7f-9ae2-dd868e325237","Type":"ContainerStarted","Data":"fa4536532b391cf01186e45302c838b9369588eeb843d536814d76070c5b01ac"} Nov 01 21:55:45.502375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.502280 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" event={"ID":"9384ed2e-f8bf-4f7f-9ae2-dd868e325237","Type":"ContainerStarted","Data":"e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e"} Nov 01 21:55:45.538932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.538891 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" podStartSLOduration=1.5388804230000002 podStartE2EDuration="1.538880423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:45.536177453 +0000 UTC m=+699.229157940" watchObservedRunningTime="2026-11-01 21:55:45.538880423 +0000 UTC m=+699.231860910" Nov 01 21:55:45.953681 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:45.953647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:46.081097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.081061 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:46.353714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.353683 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb"] Nov 01 21:55:46.357684 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.357665 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.368634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.368612 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb"] Nov 01 21:55:46.510616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.510582 5036 generic.go:356] "Generic (PLEG): container finished" podID="9384ed2e-f8bf-4f7f-9ae2-dd868e325237" containerID="fa4536532b391cf01186e45302c838b9369588eeb843d536814d76070c5b01ac" exitCode=0 Nov 01 21:55:46.511039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.510648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" event={"ID":"9384ed2e-f8bf-4f7f-9ae2-dd868e325237","Type":"ContainerDied","Data":"fa4536532b391cf01186e45302c838b9369588eeb843d536814d76070c5b01ac"} Nov 01 21:55:46.518365 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.518341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8bs\" (UniqueName: \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.519152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.518654 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.519152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.518731 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.620352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.620267 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8bs\" (UniqueName: \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.620531 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.620377 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.620531 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.620464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.621207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.621171 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.623550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.623527 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.633855 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.633830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8bs\" (UniqueName: \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\") pod \"maas-api-revoke-sub-5f245760-vfmnb\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.672299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.672279 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:46.672619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.672602 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:46.725142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.725095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:46.822766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.822732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\") pod \"6d8d920b-088b-4180-9d1b-91741afa7c7a\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " Nov 01 21:55:46.822914 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.822885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\" (UniqueName: \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\") pod \"6d8d920b-088b-4180-9d1b-91741afa7c7a\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " Nov 01 21:55:46.822993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.822932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\") pod \"6d8d920b-088b-4180-9d1b-91741afa7c7a\" (UID: \"6d8d920b-088b-4180-9d1b-91741afa7c7a\") " Nov 01 21:55:46.823176 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.823145 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca" pod "6d8d920b-088b-4180-9d1b-91741afa7c7a" (UID: "6d8d920b-088b-4180-9d1b-91741afa7c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:46.823413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.823389 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:46.825504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.825472 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca" pod "6d8d920b-088b-4180-9d1b-91741afa7c7a" (UID: "6d8d920b-088b-4180-9d1b-91741afa7c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:46.825592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.825484 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9" pod "6d8d920b-088b-4180-9d1b-91741afa7c7a" (UID: "6d8d920b-088b-4180-9d1b-91741afa7c7a"). InnerVolumeSpecName "kube-api-access-hkcs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:46.925045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.924968 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcs9\" (UniqueName: \"kubernetes.io/projected/6d8d920b-088b-4180-9d1b-91741afa7c7a-kube-api-access-hkcs9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:46.925045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.924995 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8d920b-088b-4180-9d1b-91741afa7c7a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:46.989677 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:46.989649 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb"] Nov 01 21:55:46.990790 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:46.990765 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b463a9a_b46a_4612_9679_1d56248020d6.slice/crio-714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d WatchSource:0}: Error finding container 714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d: Status 404 returned error can't find the container with id 714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d Nov 01 21:55:47.454286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.454192 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:47.518487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.518436 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" event={"ID":"3b463a9a-b46a-4612-9679-1d56248020d6","Type":"ContainerStarted","Data":"1ac9e6a8d0585592ce8a55650eeb7b2bc7e2c18cbe47fc00c09668731ea7e1db"} Nov 01 21:55:47.518919 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.518511 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" event={"ID":"3b463a9a-b46a-4612-9679-1d56248020d6","Type":"ContainerStarted","Data":"714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d"} Nov 01 21:55:47.520064 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.520027 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" event={"ID":"6d8d920b-088b-4180-9d1b-91741afa7c7a","Type":"ContainerDied","Data":"349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d"} Nov 01 21:55:47.520201 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.520085 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="349977c2c8d78bd689633ffb617f2b486d9b690a8bd244e317591edf2640563d" Nov 01 21:55:47.520201 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.520044 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p" Nov 01 21:55:47.539408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.539355 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" podStartSLOduration=1.539337754 podStartE2EDuration="1.539337754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:47.536862296 +0000 UTC m=+701.229842783" watchObservedRunningTime="2026-11-01 21:55:47.539337754 +0000 UTC m=+701.232318241" Nov 01 21:55:47.704057 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.704031 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:47.709582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.709516 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 21:55:47.710051 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710034 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9384ed2e-f8bf-4f7f-9ae2-dd868e325237" containerName="revoke-keys" Nov 01 21:55:47.710136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710054 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9384ed2e-f8bf-4f7f-9ae2-dd868e325237" containerName="revoke-keys" Nov 01 21:55:47.710136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710075 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d8d920b-088b-4180-9d1b-91741afa7c7a" containerName="revoke-keys" Nov 01 21:55:47.710136 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710084 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8d920b-088b-4180-9d1b-91741afa7c7a" containerName="revoke-keys" Nov 01 21:55:47.710285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710236 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8d920b-088b-4180-9d1b-91741afa7c7a" containerName="revoke-keys" Nov 01 21:55:47.710285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.710261 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9384ed2e-f8bf-4f7f-9ae2-dd868e325237" containerName="revoke-keys" Nov 01 21:55:47.713428 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.713410 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.720195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.720175 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb9597-openshift-default-dockercfg-lhqww\"" Nov 01 21:55:47.728237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.727670 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 21:55:47.835831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.835807 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\") pod \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " Nov 01 21:55:47.835940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.835931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\" (UniqueName: \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\") pod \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " Nov 01 21:55:47.835985 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.835970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\") pod \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\" (UID: \"9384ed2e-f8bf-4f7f-9ae2-dd868e325237\") " Nov 01 21:55:47.836108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836129 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836159 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52f4m\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836168 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca" pod "9384ed2e-f8bf-4f7f-9ae2-dd868e325237" (UID: "9384ed2e-f8bf-4f7f-9ae2-dd868e325237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:47.836424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836399 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836469 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836593 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836570 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836673 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836739 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.836897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.836883 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:47.838472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.838414 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d" pod "9384ed2e-f8bf-4f7f-9ae2-dd868e325237" (UID: "9384ed2e-f8bf-4f7f-9ae2-dd868e325237"). InnerVolumeSpecName "kube-api-access-22w7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:47.838472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.838439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca" pod "9384ed2e-f8bf-4f7f-9ae2-dd868e325237" (UID: "9384ed2e-f8bf-4f7f-9ae2-dd868e325237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:47.938000 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.937967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938020 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938114 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938331 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938672 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938768 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938751 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938838 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.938950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.938867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.939163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939139 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.939289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939216 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.939289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939259 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52f4m\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.939401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939354 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22w7d\" (UniqueName: \"kubernetes.io/projected/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-kube-api-access-22w7d\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:47.939401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939372 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9384ed2e-f8bf-4f7f-9ae2-dd868e325237-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:47.939816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939786 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.939919 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.939852 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.941067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.941035 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.941196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.941176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.951871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.951849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:47.954034 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:47.954010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52f4m\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:48.019845 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.019821 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j"] Nov 01 21:55:48.024928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.024914 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.029215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.029034 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:48.031829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.031805 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j"] Nov 01 21:55:48.141384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.141355 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fr7\" (UniqueName: \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.141560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.141501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.141630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.141577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.243367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.243329 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.243556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.243465 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.243606 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.243561 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fr7\" (UniqueName: \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.244242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.244214 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.245907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.245860 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.256291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.256265 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fr7\" (UniqueName: \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\") pod \"maas-api-revoke-sub-7dc71139-vvq4j\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.325508 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:48.325476 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4bcf45_5fd3_4874_a205_6a9e0c377aa7.slice/crio-fc367bca209319b14ba4519d999b8f686fa0b682b81db1138d75d316042efd6e WatchSource:0}: Error finding container fc367bca209319b14ba4519d999b8f686fa0b682b81db1138d75d316042efd6e: Status 404 returned error can't find the container with id fc367bca209319b14ba4519d999b8f686fa0b682b81db1138d75d316042efd6e Nov 01 21:55:48.327977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.327953 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 21:55:48.328065 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.327978 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:48.328065 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.328040 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:48.328137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.328071 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:55:48.338119 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.338094 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:48.342008 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.341985 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:48.483873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.483835 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx"] Nov 01 21:55:48.488891 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.488866 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.500465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.497485 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx"] Nov 01 21:55:48.531528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.531232 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" event={"ID":"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7","Type":"ContainerStarted","Data":"0c5f0253ce543f34b12b69d6c38b776178a511c6edae07510a1b4f0a8a37d547"} Nov 01 21:55:48.532074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.532040 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" event={"ID":"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7","Type":"ContainerStarted","Data":"fc367bca209319b14ba4519d999b8f686fa0b682b81db1138d75d316042efd6e"} Nov 01 21:55:48.533030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.532989 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" event={"ID":"9384ed2e-f8bf-4f7f-9ae2-dd868e325237","Type":"ContainerDied","Data":"e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e"} Nov 01 21:55:48.533148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.533060 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27138d6e6047c8a8933daa8e02ff7e2c72fb51810c3327bb7bdb593e953314e" Nov 01 21:55:48.533148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.533001 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8" Nov 01 21:55:48.536953 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.536926 5036 generic.go:356] "Generic (PLEG): container finished" podID="3b463a9a-b46a-4612-9679-1d56248020d6" containerID="1ac9e6a8d0585592ce8a55650eeb7b2bc7e2c18cbe47fc00c09668731ea7e1db" exitCode=0 Nov 01 21:55:48.537083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.537067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" event={"ID":"3b463a9a-b46a-4612-9679-1d56248020d6","Type":"ContainerDied","Data":"1ac9e6a8d0585592ce8a55650eeb7b2bc7e2c18cbe47fc00c09668731ea7e1db"} Nov 01 21:55:48.561938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.561881 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podStartSLOduration=1.56186595 podStartE2EDuration="1.56186595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:48.557287349 +0000 UTC m=+702.250267834" watchObservedRunningTime="2026-11-01 21:55:48.56186595 +0000 UTC m=+702.254846438" Nov 01 21:55:48.640341 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.640314 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j"] Nov 01 21:55:48.641258 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:48.641229 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5361da4_de94_4a09_80d5_f179db6548f0.slice/crio-263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735 WatchSource:0}: Error finding container 263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735: Status 404 returned error can't find the container with id 263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735 Nov 01 21:55:48.649788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.648800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.649788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.648861 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.649788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.648945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhgkc\" (UniqueName: \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.749826 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.749784 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.750018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.749850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.750018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.749903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhgkc\" (UniqueName: \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.750772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.750747 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.752271 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.752252 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.763560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.763532 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhgkc\" (UniqueName: \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\") pod \"maas-api-revoke-sub-1e134d0c-jv2tx\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.819273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:48.819237 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:48.874585 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:55:48.874524 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:55:49.030486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.030434 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:49.038962 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.038933 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:49.039102 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.038983 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:49.148973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.148949 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx"] Nov 01 21:55:49.150436 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:49.150405 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8720da_5c65_4ddc_ac8a_0a3ab6f74d31.slice/crio-051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab WatchSource:0}: Error finding container 051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab: Status 404 returned error can't find the container with id 051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab Nov 01 21:55:49.271194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.271157 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:49.545670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.545635 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" event={"ID":"f5361da4-de94-4a09-80d5-f179db6548f0","Type":"ContainerStarted","Data":"ecdcd8bdedebd5b347d1ac9994864c38df27598107b8d8fa0abe172799219bdc"} Nov 01 21:55:49.546035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.545696 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" event={"ID":"f5361da4-de94-4a09-80d5-f179db6548f0","Type":"ContainerStarted","Data":"263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735"} Nov 01 21:55:49.547111 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.547082 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" event={"ID":"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31","Type":"ContainerStarted","Data":"051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab"} Nov 01 21:55:49.564665 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.564612 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" podStartSLOduration=1.564593935 podStartE2EDuration="1.564593935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:49.564414517 +0000 UTC m=+703.257395004" watchObservedRunningTime="2026-11-01 21:55:49.564593935 +0000 UTC m=+703.257574422" Nov 01 21:55:49.761752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.761719 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:49.861269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.861244 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\" (UniqueName: \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\") pod \"3b463a9a-b46a-4612-9679-1d56248020d6\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " Nov 01 21:55:49.861391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.861339 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\") pod \"3b463a9a-b46a-4612-9679-1d56248020d6\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " Nov 01 21:55:49.861391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.861375 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\") pod \"3b463a9a-b46a-4612-9679-1d56248020d6\" (UID: \"3b463a9a-b46a-4612-9679-1d56248020d6\") " Nov 01 21:55:49.861803 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.861774 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca" pod "3b463a9a-b46a-4612-9679-1d56248020d6" (UID: "3b463a9a-b46a-4612-9679-1d56248020d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:49.863693 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.863665 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca" pod "3b463a9a-b46a-4612-9679-1d56248020d6" (UID: "3b463a9a-b46a-4612-9679-1d56248020d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:49.863860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.863834 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs" pod "3b463a9a-b46a-4612-9679-1d56248020d6" (UID: "3b463a9a-b46a-4612-9679-1d56248020d6"). InnerVolumeSpecName "kube-api-access-6m8bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:49.962949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.962917 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8bs\" (UniqueName: \"kubernetes.io/projected/3b463a9a-b46a-4612-9679-1d56248020d6-kube-api-access-6m8bs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:49.962949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.962947 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:49.962949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:49.962957 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b463a9a-b46a-4612-9679-1d56248020d6-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:50.029411 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.029380 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:50.029581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.029433 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:50.234245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.234207 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:50.554581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.554490 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" Nov 01 21:55:50.554581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.554516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb" event={"ID":"3b463a9a-b46a-4612-9679-1d56248020d6","Type":"ContainerDied","Data":"714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d"} Nov 01 21:55:50.554581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.554579 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714ba87a26624ac9ebfa4bec05d7154d5ce8d11e9af4e1300b42de2047ea093d" Nov 01 21:55:50.556213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.556187 5036 generic.go:356] "Generic (PLEG): container finished" podID="f5361da4-de94-4a09-80d5-f179db6548f0" containerID="ecdcd8bdedebd5b347d1ac9994864c38df27598107b8d8fa0abe172799219bdc" exitCode=0 Nov 01 21:55:50.556342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.556278 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" event={"ID":"f5361da4-de94-4a09-80d5-f179db6548f0","Type":"ContainerDied","Data":"ecdcd8bdedebd5b347d1ac9994864c38df27598107b8d8fa0abe172799219bdc"} Nov 01 21:55:50.558171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.558147 5036 generic.go:356] "Generic (PLEG): container finished" podID="ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" containerID="4037db95ad47f98c10b7745fc3a244a9aac10f4e4b89bc2c9412e64e24a76ab7" exitCode=0 Nov 01 21:55:50.558308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:50.558229 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" event={"ID":"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31","Type":"ContainerDied","Data":"4037db95ad47f98c10b7745fc3a244a9aac10f4e4b89bc2c9412e64e24a76ab7"} Nov 01 21:55:51.029733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.029686 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:51.029911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.029749 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:51.050148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.050109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:51.769425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.769364 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:51.846087 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.846054 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:51.860033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.860005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:51.927996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.927963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\") pod \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " Nov 01 21:55:51.928337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.928309 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca" pod "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" (UID: "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:51.928489 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.928426 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\") pod \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " Nov 01 21:55:51.928846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.928828 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\" (UniqueName: \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\") pod \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\" (UID: \"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31\") " Nov 01 21:55:51.929530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.929511 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:51.930907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.930880 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca" pod "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" (UID: "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:51.931200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:51.931178 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc" pod "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" (UID: "ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31"). InnerVolumeSpecName "kube-api-access-zhgkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:52.030271 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030242 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\") pod \"f5361da4-de94-4a09-80d5-f179db6548f0\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " Nov 01 21:55:52.030455 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030355 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:52.030455 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030392 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\" (UniqueName: \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\") pod \"f5361da4-de94-4a09-80d5-f179db6548f0\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " Nov 01 21:55:52.030455 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030408 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:52.030639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030469 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\") pod \"f5361da4-de94-4a09-80d5-f179db6548f0\" (UID: \"f5361da4-de94-4a09-80d5-f179db6548f0\") " Nov 01 21:55:52.030727 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.030700 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca" pod "f5361da4-de94-4a09-80d5-f179db6548f0" (UID: "f5361da4-de94-4a09-80d5-f179db6548f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:52.032581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.032537 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:52.032689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.032593 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:52.032689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.032613 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhgkc\" (UniqueName: \"kubernetes.io/projected/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31-kube-api-access-zhgkc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:52.043676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.043644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca" pod "f5361da4-de94-4a09-80d5-f179db6548f0" (UID: "f5361da4-de94-4a09-80d5-f179db6548f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:52.045356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.045325 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7" pod "f5361da4-de94-4a09-80d5-f179db6548f0" (UID: "f5361da4-de94-4a09-80d5-f179db6548f0"). InnerVolumeSpecName "kube-api-access-p7fr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:52.133433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.133357 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7fr7\" (UniqueName: \"kubernetes.io/projected/f5361da4-de94-4a09-80d5-f179db6548f0-kube-api-access-p7fr7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:52.133433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.133388 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5361da4-de94-4a09-80d5-f179db6548f0-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:52.572422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.572393 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" Nov 01 21:55:52.572614 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.572425 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j" event={"ID":"f5361da4-de94-4a09-80d5-f179db6548f0","Type":"ContainerDied","Data":"263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735"} Nov 01 21:55:52.572614 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.572511 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263e674fb3287934f96df085b624948b12fa5d8b17df43b304ec14e41bd36735" Nov 01 21:55:52.573815 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.573793 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" Nov 01 21:55:52.573815 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.573798 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx" event={"ID":"ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31","Type":"ContainerDied","Data":"051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab"} Nov 01 21:55:52.573976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.573840 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="051fdf0a2d1c7c39104e0548f8efa5ca25081f68587aa1479140575befac33ab" Nov 01 21:55:52.663956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:52.663926 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:53.030167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:53.030133 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:53.030566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:53.030200 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:53.557731 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:53.557687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:54.029688 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.029660 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:55:54.029859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.029726 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 21:55:54.604467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.604062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.721505 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq"] Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722321 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f5361da4-de94-4a09-80d5-f179db6548f0" containerName="revoke-keys" Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722343 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5361da4-de94-4a09-80d5-f179db6548f0" containerName="revoke-keys" Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722357 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" containerName="revoke-keys" Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722365 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" containerName="revoke-keys" Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722399 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3b463a9a-b46a-4612-9679-1d56248020d6" containerName="revoke-keys" Nov 01 21:55:54.722467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722408 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b463a9a-b46a-4612-9679-1d56248020d6" containerName="revoke-keys" Nov 01 21:55:54.722946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722554 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5361da4-de94-4a09-80d5-f179db6548f0" containerName="revoke-keys" Nov 01 21:55:54.722946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722569 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" containerName="revoke-keys" Nov 01 21:55:54.722946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.722586 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b463a9a-b46a-4612-9679-1d56248020d6" containerName="revoke-keys" Nov 01 21:55:54.762380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.760680 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq"] Nov 01 21:55:54.762380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.760805 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.762739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.762715 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.762972 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.762951 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.763152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.763134 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8f7\" (UniqueName: \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.765595 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.765571 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:55:54.766786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.766770 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:55:54.766988 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.766977 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:55:54.767062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.766990 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:55:54.864059 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.863961 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.864059 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.864023 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8f7\" (UniqueName: \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.864307 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.864095 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.865159 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.865133 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.866776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.866753 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:54.905979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:54.905955 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8f7\" (UniqueName: \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\") pod \"maas-api-revoke-sub-5c9158db-hhcsq\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:55.077366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.077333 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:55.377828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.377800 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq"] Nov 01 21:55:55.380073 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:55.380044 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b6ad9e_6692_4133_bbcb_88dfadad1578.slice/crio-9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37 WatchSource:0}: Error finding container 9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37: Status 404 returned error can't find the container with id 9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37 Nov 01 21:55:55.497877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.497840 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:55.646957 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.646918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" event={"ID":"84b6ad9e-6692-4133-bbcb-88dfadad1578","Type":"ContainerStarted","Data":"9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37"} Nov 01 21:55:55.921980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.921951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:55.922906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.922770 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:55.923898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:55.923874 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 21:55:56.281038 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.281003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:56.415469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.413792 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:56.568279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.568196 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc"] Nov 01 21:55:56.572160 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.572137 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.581030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.580999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trffl\" (UniqueName: \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.581230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.581205 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.581302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.581287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.582635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.582608 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc"] Nov 01 21:55:56.654922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.654893 5036 generic.go:356] "Generic (PLEG): container finished" podID="84b6ad9e-6692-4133-bbcb-88dfadad1578" containerID="db5816b10a7f032ccb4b41564f6dbb61a25ce5ba416151a38ce29b87965712dd" exitCode=0 Nov 01 21:55:56.655296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.654981 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" event={"ID":"84b6ad9e-6692-4133-bbcb-88dfadad1578","Type":"ContainerDied","Data":"db5816b10a7f032ccb4b41564f6dbb61a25ce5ba416151a38ce29b87965712dd"} Nov 01 21:55:56.682374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.682349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.682533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.682414 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.682533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.682491 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trffl\" (UniqueName: \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.683264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.683239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.684902 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.684878 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.695387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.695365 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trffl\" (UniqueName: \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\") pod \"maas-api-revoke-sub-10d8fb5e-h8mkc\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:56.888567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:56.888482 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:57.175200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.175171 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc"] Nov 01 21:55:57.175471 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:55:57.175430 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c00f306_c89e_42b2_8a67_82bdffcae3a7.slice/crio-9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51 WatchSource:0}: Error finding container 9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51: Status 404 returned error can't find the container with id 9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51 Nov 01 21:55:57.234024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.233995 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:57.662518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.662478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" event={"ID":"9c00f306-c89e-42b2-8a67-82bdffcae3a7","Type":"ContainerStarted","Data":"5f1b3091dfd771795ba84063351445b3a92f8aad8817aaa312b47162ed0446fa"} Nov 01 21:55:57.662928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.662539 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" event={"ID":"9c00f306-c89e-42b2-8a67-82bdffcae3a7","Type":"ContainerStarted","Data":"9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51"} Nov 01 21:55:57.683994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.683946 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" podStartSLOduration=1.6839314920000001 podStartE2EDuration="1.683931492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:57.681093918 +0000 UTC m=+711.374074407" watchObservedRunningTime="2026-11-01 21:55:57.683931492 +0000 UTC m=+711.376911977" Nov 01 21:55:57.804116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.804089 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:57.894034 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.894003 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\") pod \"84b6ad9e-6692-4133-bbcb-88dfadad1578\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " Nov 01 21:55:57.894196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.894065 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\") pod \"84b6ad9e-6692-4133-bbcb-88dfadad1578\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " Nov 01 21:55:57.894196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.894126 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\" (UniqueName: \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\") pod \"84b6ad9e-6692-4133-bbcb-88dfadad1578\" (UID: \"84b6ad9e-6692-4133-bbcb-88dfadad1578\") " Nov 01 21:55:57.894425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.894404 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca" pod "84b6ad9e-6692-4133-bbcb-88dfadad1578" (UID: "84b6ad9e-6692-4133-bbcb-88dfadad1578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:57.894543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.894521 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:57.896145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.896122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7" pod "84b6ad9e-6692-4133-bbcb-88dfadad1578" (UID: "84b6ad9e-6692-4133-bbcb-88dfadad1578"). InnerVolumeSpecName "kube-api-access-7v8f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:57.896221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.896196 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca" pod "84b6ad9e-6692-4133-bbcb-88dfadad1578" (UID: "84b6ad9e-6692-4133-bbcb-88dfadad1578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:57.943667 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.943635 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:57.995494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.995464 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b6ad9e-6692-4133-bbcb-88dfadad1578-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:57.995494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:57.995493 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8f7\" (UniqueName: \"kubernetes.io/projected/84b6ad9e-6692-4133-bbcb-88dfadad1578-kube-api-access-7v8f7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:58.641475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.641419 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:58.670802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.670769 5036 generic.go:356] "Generic (PLEG): container finished" podID="9c00f306-c89e-42b2-8a67-82bdffcae3a7" containerID="5f1b3091dfd771795ba84063351445b3a92f8aad8817aaa312b47162ed0446fa" exitCode=0 Nov 01 21:55:58.671260 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.670849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" event={"ID":"9c00f306-c89e-42b2-8a67-82bdffcae3a7","Type":"ContainerDied","Data":"5f1b3091dfd771795ba84063351445b3a92f8aad8817aaa312b47162ed0446fa"} Nov 01 21:55:58.672328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.672305 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" Nov 01 21:55:58.672502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.672303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq" event={"ID":"84b6ad9e-6692-4133-bbcb-88dfadad1578","Type":"ContainerDied","Data":"9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37"} Nov 01 21:55:58.672502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:58.672427 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d4f8b1989e1546c32a3db54963c5184bd8dfbb07f33d08147cd243552258d37" Nov 01 21:55:59.397972 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.397935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:59.831231 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.831204 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:55:59.916510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.916478 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\") pod \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " Nov 01 21:55:59.916675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.916635 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\") pod \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " Nov 01 21:55:59.916736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.916683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\" (UniqueName: \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\") pod \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\" (UID: \"9c00f306-c89e-42b2-8a67-82bdffcae3a7\") " Nov 01 21:55:59.916859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.916829 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca" pod "9c00f306-c89e-42b2-8a67-82bdffcae3a7" (UID: "9c00f306-c89e-42b2-8a67-82bdffcae3a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:59.917019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.916990 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:59.918821 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.918793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl" pod "9c00f306-c89e-42b2-8a67-82bdffcae3a7" (UID: "9c00f306-c89e-42b2-8a67-82bdffcae3a7"). InnerVolumeSpecName "kube-api-access-trffl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:59.918821 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.918795 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca" pod "9c00f306-c89e-42b2-8a67-82bdffcae3a7" (UID: "9c00f306-c89e-42b2-8a67-82bdffcae3a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:59.942710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.942653 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf"] Nov 01 21:55:59.943191 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943174 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9c00f306-c89e-42b2-8a67-82bdffcae3a7" containerName="revoke-keys" Nov 01 21:55:59.943272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943194 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c00f306-c89e-42b2-8a67-82bdffcae3a7" containerName="revoke-keys" Nov 01 21:55:59.943272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943229 5036 cpu_manager.go:414] "containerMap: removing container" podUID="84b6ad9e-6692-4133-bbcb-88dfadad1578" containerName="revoke-keys" Nov 01 21:55:59.943272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943238 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b6ad9e-6692-4133-bbcb-88dfadad1578" containerName="revoke-keys" Nov 01 21:55:59.943421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943394 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b6ad9e-6692-4133-bbcb-88dfadad1578" containerName="revoke-keys" Nov 01 21:55:59.943421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.943414 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c00f306-c89e-42b2-8a67-82bdffcae3a7" containerName="revoke-keys" Nov 01 21:55:59.946798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.946779 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:55:59.954411 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:55:59.954381 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf"] Nov 01 21:56:00.018041 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.018011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.018209 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.018135 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.018209 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.018194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwdr\" (UniqueName: \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.018338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.018325 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c00f306-c89e-42b2-8a67-82bdffcae3a7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:00.018380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.018341 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trffl\" (UniqueName: \"kubernetes.io/projected/9c00f306-c89e-42b2-8a67-82bdffcae3a7-kube-api-access-trffl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:00.119894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.119851 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.120067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.119906 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwdr\" (UniqueName: \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.120067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.120040 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.120791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.120761 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.122557 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.122530 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.123415 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.123388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:00.136068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.136036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwdr\" (UniqueName: \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\") pod \"maas-api-revoke-sub-fd6c32b6-scddf\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.276801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.276773 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:00.558409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.558381 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf"] Nov 01 21:56:00.558733 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:00.558712 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dcbe9e_ef62_45fd_9ac3_e622a5059e6c.slice/crio-7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc WatchSource:0}: Error finding container 7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc: Status 404 returned error can't find the container with id 7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc Nov 01 21:56:00.686177 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.686153 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" Nov 01 21:56:00.686407 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.686155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc" event={"ID":"9c00f306-c89e-42b2-8a67-82bdffcae3a7","Type":"ContainerDied","Data":"9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51"} Nov 01 21:56:00.686407 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.686244 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c83786a40fd9519fbe46ddb904cedb6738f732ff38f26ad922e2dd982b99d51" Nov 01 21:56:00.687636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.687609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" event={"ID":"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c","Type":"ContainerStarted","Data":"7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc"} Nov 01 21:56:00.926646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:00.926615 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:01.582715 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.582683 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4"] Nov 01 21:56:01.586517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.586496 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.597018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.596730 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4"] Nov 01 21:56:01.631714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.631688 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:01.636974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.636904 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.637086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.636970 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.637086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.637063 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz49c\" (UniqueName: \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.698137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.698102 5036 generic.go:356] "Generic (PLEG): container finished" podID="d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" containerID="240c24d039f59f6b54cd635ac23f4a3bbc562ef5be4afc085f77bf39d82fdb37" exitCode=0 Nov 01 21:56:01.698314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.698179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" event={"ID":"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c","Type":"ContainerDied","Data":"240c24d039f59f6b54cd635ac23f4a3bbc562ef5be4afc085f77bf39d82fdb37"} Nov 01 21:56:01.738012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.737983 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.738189 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.738040 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.738189 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.738070 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz49c\" (UniqueName: \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.738940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.738914 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.740410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.740387 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.750578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.750557 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz49c\" (UniqueName: \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\") pod \"maas-api-revoke-sub-36b1b603-8rws4\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:01.903658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:01.903576 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:02.075635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.075601 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp"] Nov 01 21:56:02.080728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.080709 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.097016 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.092233 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp"] Nov 01 21:56:02.141650 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.141620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.141830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.141808 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.141883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.141871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cnt\" (UniqueName: \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.188974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.188944 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4"] Nov 01 21:56:02.189296 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:02.189267 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81363b1_f85c_4d3e_ac08_62aab7b7095d.slice/crio-8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2 WatchSource:0}: Error finding container 8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2: Status 404 returned error can't find the container with id 8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2 Nov 01 21:56:02.243090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.243061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.243238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.243221 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.243294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.243282 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cnt\" (UniqueName: \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.243836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.243810 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.245376 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.245353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.255711 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.255691 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cnt\" (UniqueName: \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\") pod \"maas-api-revoke-sub-9876f6fe-wb5hp\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.404909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.404869 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:02.494834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.494806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:02.685095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.685067 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp"] Nov 01 21:56:02.685658 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:02.685633 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66478636_93a1_4486_b7bd_65b95ef1d150.slice/crio-8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27 WatchSource:0}: Error finding container 8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27: Status 404 returned error can't find the container with id 8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27 Nov 01 21:56:02.706668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.706634 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" event={"ID":"f81363b1-f85c-4d3e-ac08-62aab7b7095d","Type":"ContainerStarted","Data":"012818c82f4fd27e846bb235942c6965dcf10673e950a6459cbbe19e9fcd9071"} Nov 01 21:56:02.706935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.706848 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" event={"ID":"f81363b1-f85c-4d3e-ac08-62aab7b7095d","Type":"ContainerStarted","Data":"8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2"} Nov 01 21:56:02.708357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.708304 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" event={"ID":"66478636-93a1-4486-b7bd-65b95ef1d150","Type":"ContainerStarted","Data":"8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27"} Nov 01 21:56:02.727554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.727514 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" podStartSLOduration=1.7274995039999999 podStartE2EDuration="1.727499504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:02.724605988 +0000 UTC m=+716.417586475" watchObservedRunningTime="2026-11-01 21:56:02.727499504 +0000 UTC m=+716.420479990" Nov 01 21:56:02.835846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.835818 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:02.950830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.950799 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\") pod \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " Nov 01 21:56:02.950992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.950909 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\") pod \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " Nov 01 21:56:02.951066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.951014 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\" (UniqueName: \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\") pod \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\" (UID: \"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c\") " Nov 01 21:56:02.951357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.951324 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca" pod "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" (UID: "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:02.953031 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.952999 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca" pod "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" (UID: "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:02.953123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:02.953043 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr" pod "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" (UID: "d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c"). InnerVolumeSpecName "kube-api-access-9vwdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:03.052010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.051979 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwdr\" (UniqueName: \"kubernetes.io/projected/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-kube-api-access-9vwdr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:03.052010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.052007 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:03.052180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.052018 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:03.220589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.220510 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:03.718659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.718629 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" Nov 01 21:56:03.718659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.718641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf" event={"ID":"d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c","Type":"ContainerDied","Data":"7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc"} Nov 01 21:56:03.718930 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.718693 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7510cde8d380fab6b4d46bc0c05255792e57669940dc1f8e6c24e47325bde3dc" Nov 01 21:56:03.720506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.720485 5036 generic.go:356] "Generic (PLEG): container finished" podID="f81363b1-f85c-4d3e-ac08-62aab7b7095d" containerID="012818c82f4fd27e846bb235942c6965dcf10673e950a6459cbbe19e9fcd9071" exitCode=0 Nov 01 21:56:03.720635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.720574 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" event={"ID":"f81363b1-f85c-4d3e-ac08-62aab7b7095d","Type":"ContainerDied","Data":"012818c82f4fd27e846bb235942c6965dcf10673e950a6459cbbe19e9fcd9071"} Nov 01 21:56:03.722435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.722404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" event={"ID":"66478636-93a1-4486-b7bd-65b95ef1d150","Type":"ContainerStarted","Data":"d3619c56333caaabb4d099ddcc95a0bc2efc0b853225e52c7e9506e93db55589"} Nov 01 21:56:03.763326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:03.763271 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" podStartSLOduration=1.76325379 podStartE2EDuration="1.76325379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:03.760140197 +0000 UTC m=+717.453120684" watchObservedRunningTime="2026-11-01 21:56:03.76325379 +0000 UTC m=+717.456234277" Nov 01 21:56:04.016076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.015967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:04.731431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.731399 5036 generic.go:356] "Generic (PLEG): container finished" podID="66478636-93a1-4486-b7bd-65b95ef1d150" containerID="d3619c56333caaabb4d099ddcc95a0bc2efc0b853225e52c7e9506e93db55589" exitCode=0 Nov 01 21:56:04.731856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.731489 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" event={"ID":"66478636-93a1-4486-b7bd-65b95ef1d150","Type":"ContainerDied","Data":"d3619c56333caaabb4d099ddcc95a0bc2efc0b853225e52c7e9506e93db55589"} Nov 01 21:56:04.828501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.828405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:04.905787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.905760 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:04.972295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.972268 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\") pod \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " Nov 01 21:56:04.972435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.972386 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\" (UniqueName: \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\") pod \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " Nov 01 21:56:04.972435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.972427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\") pod \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\" (UID: \"f81363b1-f85c-4d3e-ac08-62aab7b7095d\") " Nov 01 21:56:04.972830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.972804 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca" pod "f81363b1-f85c-4d3e-ac08-62aab7b7095d" (UID: "f81363b1-f85c-4d3e-ac08-62aab7b7095d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:04.974393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.974365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca" pod "f81363b1-f85c-4d3e-ac08-62aab7b7095d" (UID: "f81363b1-f85c-4d3e-ac08-62aab7b7095d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:04.974510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:04.974414 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c" pod "f81363b1-f85c-4d3e-ac08-62aab7b7095d" (UID: "f81363b1-f85c-4d3e-ac08-62aab7b7095d"). InnerVolumeSpecName "kube-api-access-xz49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:05.073727 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.073701 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.073727 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.073727 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz49c\" (UniqueName: \"kubernetes.io/projected/f81363b1-f85c-4d3e-ac08-62aab7b7095d-kube-api-access-xz49c\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.073877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.073739 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81363b1-f85c-4d3e-ac08-62aab7b7095d-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.551646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.551614 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:05.739739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.739709 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" Nov 01 21:56:05.740214 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.739753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4" event={"ID":"f81363b1-f85c-4d3e-ac08-62aab7b7095d","Type":"ContainerDied","Data":"8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2"} Nov 01 21:56:05.740214 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.739802 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cfd320fba2e5e7701761b57bcc80888a763837d849cb0c04d8e32fa318de2e2" Nov 01 21:56:05.882582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.882551 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:05.984375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.984336 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\") pod \"66478636-93a1-4486-b7bd-65b95ef1d150\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " Nov 01 21:56:05.984553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.984468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\") pod \"66478636-93a1-4486-b7bd-65b95ef1d150\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " Nov 01 21:56:05.984553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.984533 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\" (UniqueName: \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\") pod \"66478636-93a1-4486-b7bd-65b95ef1d150\" (UID: \"66478636-93a1-4486-b7bd-65b95ef1d150\") " Nov 01 21:56:05.984900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.984870 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca" pod "66478636-93a1-4486-b7bd-65b95ef1d150" (UID: "66478636-93a1-4486-b7bd-65b95ef1d150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:05.985252 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.985228 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.987127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.987099 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca" pod "66478636-93a1-4486-b7bd-65b95ef1d150" (UID: "66478636-93a1-4486-b7bd-65b95ef1d150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:05.987206 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:05.987129 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt" pod "66478636-93a1-4486-b7bd-65b95ef1d150" (UID: "66478636-93a1-4486-b7bd-65b95ef1d150"). InnerVolumeSpecName "kube-api-access-47cnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:06.087038 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.086962 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66478636-93a1-4486-b7bd-65b95ef1d150-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:06.087038 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.087001 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cnt\" (UniqueName: \"kubernetes.io/projected/66478636-93a1-4486-b7bd-65b95ef1d150-kube-api-access-47cnt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:06.109884 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.109847 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:56:06.349489 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.349364 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:06.529274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529242 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd"] Nov 01 21:56:06.529842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529822 5036 cpu_manager.go:414] "containerMap: removing container" podUID="66478636-93a1-4486-b7bd-65b95ef1d150" containerName="revoke-keys" Nov 01 21:56:06.529842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529843 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66478636-93a1-4486-b7bd-65b95ef1d150" containerName="revoke-keys" Nov 01 21:56:06.530018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529871 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f81363b1-f85c-4d3e-ac08-62aab7b7095d" containerName="revoke-keys" Nov 01 21:56:06.530018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529879 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81363b1-f85c-4d3e-ac08-62aab7b7095d" containerName="revoke-keys" Nov 01 21:56:06.530018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529923 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" containerName="revoke-keys" Nov 01 21:56:06.530018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.529931 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" containerName="revoke-keys" Nov 01 21:56:06.530203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.530061 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" containerName="revoke-keys" Nov 01 21:56:06.530203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.530078 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="66478636-93a1-4486-b7bd-65b95ef1d150" containerName="revoke-keys" Nov 01 21:56:06.530203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.530092 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81363b1-f85c-4d3e-ac08-62aab7b7095d" containerName="revoke-keys" Nov 01 21:56:06.533691 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.533670 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.543992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.543971 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd"] Nov 01 21:56:06.591797 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.591748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.591947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.591854 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmt7\" (UniqueName: \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.592033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.592009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.693605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.693508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.693605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.693557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmt7\" (UniqueName: \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.693818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.693626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.694394 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.694361 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.696060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.696038 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.705687 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.705662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmt7\" (UniqueName: \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\") pod \"maas-api-revoke-sub-aed9728e-m9zsd\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:06.748732 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.748706 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" Nov 01 21:56:06.749211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.748703 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp" event={"ID":"66478636-93a1-4486-b7bd-65b95ef1d150","Type":"ContainerDied","Data":"8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27"} Nov 01 21:56:06.749211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.748807 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a044ab5fd6f970f838c7cca9f62ddb8d72242191e4ec8aaa06df03700c92c27" Nov 01 21:56:06.852255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:06.852209 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:07.423469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:07.405341 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd"] Nov 01 21:56:07.769129 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:07.769095 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" event={"ID":"fb3b8031-d0f9-4e4a-b471-6529f62c4502","Type":"ContainerStarted","Data":"9644fa6a6238f97d9ac37fad54c5d96cac560a13d088e020dafb3b88b9ea388e"} Nov 01 21:56:08.243575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:08.243524 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:08.812499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:08.811946 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" event={"ID":"fb3b8031-d0f9-4e4a-b471-6529f62c4502","Type":"ContainerStarted","Data":"a8681a8108aec5a65669796ea37047cce759bb1d4b06ab45a811f452d77ca49b"} Nov 01 21:56:09.801475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:09.798948 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:09.827470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:09.827199 5036 generic.go:356] "Generic (PLEG): container finished" podID="fb3b8031-d0f9-4e4a-b471-6529f62c4502" containerID="a8681a8108aec5a65669796ea37047cce759bb1d4b06ab45a811f452d77ca49b" exitCode=0 Nov 01 21:56:09.827971 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:09.827501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" event={"ID":"fb3b8031-d0f9-4e4a-b471-6529f62c4502","Type":"ContainerDied","Data":"a8681a8108aec5a65669796ea37047cce759bb1d4b06ab45a811f452d77ca49b"} Nov 01 21:56:10.927799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:10.927701 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:11.018358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.018327 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:11.080605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.080571 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:11.161948 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.161853 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\" (UniqueName: \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\") pod \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " Nov 01 21:56:11.162173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.161965 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\") pod \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " Nov 01 21:56:11.162173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.162065 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\") pod \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\" (UID: \"fb3b8031-d0f9-4e4a-b471-6529f62c4502\") " Nov 01 21:56:11.180586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.168105 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca" pod "fb3b8031-d0f9-4e4a-b471-6529f62c4502" (UID: "fb3b8031-d0f9-4e4a-b471-6529f62c4502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:11.187738 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.187696 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca" pod "fb3b8031-d0f9-4e4a-b471-6529f62c4502" (UID: "fb3b8031-d0f9-4e4a-b471-6529f62c4502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:11.189096 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.189071 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7" pod "fb3b8031-d0f9-4e4a-b471-6529f62c4502" (UID: "fb3b8031-d0f9-4e4a-b471-6529f62c4502"). InnerVolumeSpecName "kube-api-access-rkmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:11.263421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.263388 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:11.263421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.263416 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b8031-d0f9-4e4a-b471-6529f62c4502-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:11.263421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.263426 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmt7\" (UniqueName: \"kubernetes.io/projected/fb3b8031-d0f9-4e4a-b471-6529f62c4502-kube-api-access-rkmt7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:11.802958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.802912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:11.848343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.848301 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" event={"ID":"fb3b8031-d0f9-4e4a-b471-6529f62c4502","Type":"ContainerDied","Data":"9644fa6a6238f97d9ac37fad54c5d96cac560a13d088e020dafb3b88b9ea388e"} Nov 01 21:56:11.848533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.848366 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9644fa6a6238f97d9ac37fad54c5d96cac560a13d088e020dafb3b88b9ea388e" Nov 01 21:56:11.848533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.848478 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd" Nov 01 21:56:11.922205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.922171 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h"] Nov 01 21:56:11.922851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.922822 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fb3b8031-d0f9-4e4a-b471-6529f62c4502" containerName="revoke-keys" Nov 01 21:56:11.922851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.922845 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3b8031-d0f9-4e4a-b471-6529f62c4502" containerName="revoke-keys" Nov 01 21:56:11.923087 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.923005 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3b8031-d0f9-4e4a-b471-6529f62c4502" containerName="revoke-keys" Nov 01 21:56:11.926769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.926744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:11.930708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.930682 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:56:11.931092 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.930913 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:56:11.931981 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.931961 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:56:11.932097 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.932026 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:56:11.942467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.940320 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h"] Nov 01 21:56:11.970827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.970802 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnnl\" (UniqueName: \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:11.970956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.970929 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:11.971000 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:11.970953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.071802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.071723 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.071802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.071764 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.072028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.071837 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnnl\" (UniqueName: \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.072488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.072435 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.074195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.074172 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.084584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.084557 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnnl\" (UniqueName: \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\") pod \"maas-api-revoke-sub-b2e5f2c4-8sd9h\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.244218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.244181 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:12.441607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.441524 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm"] Nov 01 21:56:12.448027 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.447999 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.463676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.463649 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm"] Nov 01 21:56:12.533850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.533821 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h"] Nov 01 21:56:12.534387 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:12.534363 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f56d51_6e01_4c34_8e19_2fd193b3da6f.slice/crio-ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab WatchSource:0}: Error finding container ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab: Status 404 returned error can't find the container with id ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab Nov 01 21:56:12.575759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.575735 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhth\" (UniqueName: \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.575954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.575870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.575954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.575933 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.676926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.676895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.677135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.677000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.677135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.677076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhth\" (UniqueName: \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.678042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.678010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.679674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.679645 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.689455 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.689420 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhth\" (UniqueName: \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\") pod \"maas-api-revoke-sub-2de1fe96-plbpm\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.755288 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.755202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:12.766615 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.766588 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:12.862290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.861799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" event={"ID":"a1f56d51-6e01-4c34-8e19-2fd193b3da6f","Type":"ContainerStarted","Data":"fb04f8b4c3933be736d2b08f6e1d3a44c38aa7174f93b76330e6bd3a9d0deb53"} Nov 01 21:56:12.862290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.862100 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" event={"ID":"a1f56d51-6e01-4c34-8e19-2fd193b3da6f","Type":"ContainerStarted","Data":"ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab"} Nov 01 21:56:12.882531 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:12.882483 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" podStartSLOduration=1.882439325 podStartE2EDuration="1.882439325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:12.879850203 +0000 UTC m=+726.572830690" watchObservedRunningTime="2026-11-01 21:56:12.882439325 +0000 UTC m=+726.575419853" Nov 01 21:56:13.069977 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:13.069947 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4892cee_16ca_4fa2_a0a8_e2178271a588.slice/crio-a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98 WatchSource:0}: Error finding container a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98: Status 404 returned error can't find the container with id a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98 Nov 01 21:56:13.070903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.070870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm"] Nov 01 21:56:13.535708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.535672 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:13.870906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.870873 5036 generic.go:356] "Generic (PLEG): container finished" podID="a1f56d51-6e01-4c34-8e19-2fd193b3da6f" containerID="fb04f8b4c3933be736d2b08f6e1d3a44c38aa7174f93b76330e6bd3a9d0deb53" exitCode=0 Nov 01 21:56:13.871114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.870963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" event={"ID":"a1f56d51-6e01-4c34-8e19-2fd193b3da6f","Type":"ContainerDied","Data":"fb04f8b4c3933be736d2b08f6e1d3a44c38aa7174f93b76330e6bd3a9d0deb53"} Nov 01 21:56:13.872599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.872577 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" event={"ID":"a4892cee-16ca-4fa2-a0a8-e2178271a588","Type":"ContainerStarted","Data":"f4eece0212aa90796cb9aad28672431e1bc375e2f86fb6eaae690a6fb05e1a8e"} Nov 01 21:56:13.872715 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.872618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" event={"ID":"a4892cee-16ca-4fa2-a0a8-e2178271a588","Type":"ContainerStarted","Data":"a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98"} Nov 01 21:56:13.955567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.955538 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf"] Nov 01 21:56:13.959348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.959319 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:13.972394 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.970904 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf"] Nov 01 21:56:13.972607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:13.972554 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" podStartSLOduration=1.972537468 podStartE2EDuration="1.972537468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:13.970375665 +0000 UTC m=+727.663356152" watchObservedRunningTime="2026-11-01 21:56:13.972537468 +0000 UTC m=+727.665517955" Nov 01 21:56:14.095384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.095326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.095806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.095387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfd7\" (UniqueName: \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.095806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.095579 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.197109 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.197030 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.197266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.197138 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.197266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.197159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfd7\" (UniqueName: \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.197917 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.197896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.199642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.199618 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.209110 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.209068 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfd7\" (UniqueName: \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\") pod \"maas-api-revoke-sub-ccf8d366-hctzf\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.281041 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.281009 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:14.343424 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.342981 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:14.568218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.568187 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf"] Nov 01 21:56:14.568816 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:14.568783 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d41a1_c8d1_4625_86cb_aea64f957630.slice/crio-70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076 WatchSource:0}: Error finding container 70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076: Status 404 returned error can't find the container with id 70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076 Nov 01 21:56:14.880294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.880205 5036 generic.go:356] "Generic (PLEG): container finished" podID="a4892cee-16ca-4fa2-a0a8-e2178271a588" containerID="f4eece0212aa90796cb9aad28672431e1bc375e2f86fb6eaae690a6fb05e1a8e" exitCode=0 Nov 01 21:56:14.880473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.880298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" event={"ID":"a4892cee-16ca-4fa2-a0a8-e2178271a588","Type":"ContainerDied","Data":"f4eece0212aa90796cb9aad28672431e1bc375e2f86fb6eaae690a6fb05e1a8e"} Nov 01 21:56:14.881813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:14.881786 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" event={"ID":"583d41a1-c8d1-4625-86cb-aea64f957630","Type":"ContainerStarted","Data":"70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076"} Nov 01 21:56:15.037975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.037944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:15.041256 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.041159 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:15.107163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.107135 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\") pod \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " Nov 01 21:56:15.107577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.107302 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\" (UniqueName: \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\") pod \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " Nov 01 21:56:15.107577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.107338 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\") pod \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\" (UID: \"a1f56d51-6e01-4c34-8e19-2fd193b3da6f\") " Nov 01 21:56:15.107674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.107567 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca" pod "a1f56d51-6e01-4c34-8e19-2fd193b3da6f" (UID: "a1f56d51-6e01-4c34-8e19-2fd193b3da6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:15.107800 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.107780 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:15.109387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.109361 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl" pod "a1f56d51-6e01-4c34-8e19-2fd193b3da6f" (UID: "a1f56d51-6e01-4c34-8e19-2fd193b3da6f"). InnerVolumeSpecName "kube-api-access-8bnnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:15.109761 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.109742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca" pod "a1f56d51-6e01-4c34-8e19-2fd193b3da6f" (UID: "a1f56d51-6e01-4c34-8e19-2fd193b3da6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:15.209083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.209055 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bnnl\" (UniqueName: \"kubernetes.io/projected/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-kube-api-access-8bnnl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:15.209164 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.209089 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56d51-6e01-4c34-8e19-2fd193b3da6f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:15.725151 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.725113 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:15.890516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.890456 5036 generic.go:356] "Generic (PLEG): container finished" podID="583d41a1-c8d1-4625-86cb-aea64f957630" containerID="11a34a1a936d77f2c7b64a47b1ee0f693f0470cc2360df24dcc9afb39f599dc9" exitCode=0 Nov 01 21:56:15.890516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.890477 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" event={"ID":"583d41a1-c8d1-4625-86cb-aea64f957630","Type":"ContainerDied","Data":"11a34a1a936d77f2c7b64a47b1ee0f693f0470cc2360df24dcc9afb39f599dc9"} Nov 01 21:56:15.892197 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.892148 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" event={"ID":"a1f56d51-6e01-4c34-8e19-2fd193b3da6f","Type":"ContainerDied","Data":"ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab"} Nov 01 21:56:15.892197 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.892175 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h" Nov 01 21:56:15.892197 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:15.892199 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4901667317e914062381cd9725dd5cadfbde48b38c301aad4d78de5fd72bab" Nov 01 21:56:16.057868 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.057841 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:16.119249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.119218 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\") pod \"a4892cee-16ca-4fa2-a0a8-e2178271a588\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " Nov 01 21:56:16.119632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.119298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\" (UniqueName: \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\") pod \"a4892cee-16ca-4fa2-a0a8-e2178271a588\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " Nov 01 21:56:16.119632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.119391 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\") pod \"a4892cee-16ca-4fa2-a0a8-e2178271a588\" (UID: \"a4892cee-16ca-4fa2-a0a8-e2178271a588\") " Nov 01 21:56:16.119773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.119752 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca" pod "a4892cee-16ca-4fa2-a0a8-e2178271a588" (UID: "a4892cee-16ca-4fa2-a0a8-e2178271a588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:16.121399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.121373 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca" pod "a4892cee-16ca-4fa2-a0a8-e2178271a588" (UID: "a4892cee-16ca-4fa2-a0a8-e2178271a588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:16.121836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.121813 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth" pod "a4892cee-16ca-4fa2-a0a8-e2178271a588" (UID: "a4892cee-16ca-4fa2-a0a8-e2178271a588"). InnerVolumeSpecName "kube-api-access-vwhth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:16.220963 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.220931 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhth\" (UniqueName: \"kubernetes.io/projected/a4892cee-16ca-4fa2-a0a8-e2178271a588-kube-api-access-vwhth\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:16.220963 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.220959 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:16.220963 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.220970 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4892cee-16ca-4fa2-a0a8-e2178271a588-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:16.439117 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.439022 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:16.901933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.901905 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" Nov 01 21:56:16.902400 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.902377 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm" event={"ID":"a4892cee-16ca-4fa2-a0a8-e2178271a588","Type":"ContainerDied","Data":"a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98"} Nov 01 21:56:16.902527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:16.902427 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b92eb7d3ed6792de824628c2de88430704eb4307ff379e06095fb6a881fd98" Nov 01 21:56:17.073485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.073431 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:17.131456 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.131412 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\" (UniqueName: \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\") pod \"583d41a1-c8d1-4625-86cb-aea64f957630\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " Nov 01 21:56:17.131882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.131523 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\") pod \"583d41a1-c8d1-4625-86cb-aea64f957630\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " Nov 01 21:56:17.131882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.131789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\") pod \"583d41a1-c8d1-4625-86cb-aea64f957630\" (UID: \"583d41a1-c8d1-4625-86cb-aea64f957630\") " Nov 01 21:56:17.132068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.132042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca" pod "583d41a1-c8d1-4625-86cb-aea64f957630" (UID: "583d41a1-c8d1-4625-86cb-aea64f957630"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:17.132200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.132182 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:17.134522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.134496 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7" pod "583d41a1-c8d1-4625-86cb-aea64f957630" (UID: "583d41a1-c8d1-4625-86cb-aea64f957630"). InnerVolumeSpecName "kube-api-access-zjfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:17.135548 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.135520 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca" pod "583d41a1-c8d1-4625-86cb-aea64f957630" (UID: "583d41a1-c8d1-4625-86cb-aea64f957630"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:17.233630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.233555 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583d41a1-c8d1-4625-86cb-aea64f957630-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:17.233630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.233582 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfd7\" (UniqueName: \"kubernetes.io/projected/583d41a1-c8d1-4625-86cb-aea64f957630-kube-api-access-zjfd7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:17.240245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.240209 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:17.910572 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.910544 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" Nov 01 21:56:17.910762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.910591 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf" event={"ID":"583d41a1-c8d1-4625-86cb-aea64f957630","Type":"ContainerDied","Data":"70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076"} Nov 01 21:56:17.910762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:17.910649 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a4f8ee75d2e9003bffa91aa22ec42fcd5451be1bc57c2c48e11d9ba56bb076" Nov 01 21:56:18.057398 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.057360 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:18.122131 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122086 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 21:56:18.122743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122716 5036 cpu_manager.go:414] "containerMap: removing container" podUID="583d41a1-c8d1-4625-86cb-aea64f957630" containerName="revoke-keys" Nov 01 21:56:18.122743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122743 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583d41a1-c8d1-4625-86cb-aea64f957630" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122766 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a4892cee-16ca-4fa2-a0a8-e2178271a588" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122774 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4892cee-16ca-4fa2-a0a8-e2178271a588" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122796 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a1f56d51-6e01-4c34-8e19-2fd193b3da6f" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122801 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f56d51-6e01-4c34-8e19-2fd193b3da6f" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122898 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="583d41a1-c8d1-4625-86cb-aea64f957630" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122909 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4892cee-16ca-4fa2-a0a8-e2178271a588" containerName="revoke-keys" Nov 01 21:56:18.122955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.122918 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f56d51-6e01-4c34-8e19-2fd193b3da6f" containerName="revoke-keys" Nov 01 21:56:18.126786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.126762 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.137740 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.137709 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 21:56:18.161481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.161387 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ff3e70-openshift-default-dockercfg-8tnvh\"" Nov 01 21:56:18.249255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249216 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249260 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249439 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqct\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249672 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.249986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.249792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350653 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350690 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350711 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350821 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350854 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.350915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.350888 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351015 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351108 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351083 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351217 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351147 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351217 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351217 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351182 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqct\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351222 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351351 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.351877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.351852 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.353186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.353167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.353607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.353583 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.362556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.362529 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.363522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.363502 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqct\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.446182 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.446102 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:18.742340 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:18.742307 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91faaff1_4004_4afa_8f65_2705df604913.slice/crio-b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913 WatchSource:0}: Error finding container b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913: Status 404 returned error can't find the container with id b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913 Nov 01 21:56:18.742519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.742494 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 21:56:18.744916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.744888 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:18.745009 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.744964 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:18.745009 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.744995 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:18.864879 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:56:18.863109 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:56:18.921465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.920560 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" event={"ID":"91faaff1-4004-4afa-8f65-2705df604913","Type":"ContainerStarted","Data":"0062a2239dbc4dd3099e2c6dc0668986a66f7d104830b20c03625f3461b3c246"} Nov 01 21:56:18.921465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.920638 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" event={"ID":"91faaff1-4004-4afa-8f65-2705df604913","Type":"ContainerStarted","Data":"b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913"} Nov 01 21:56:18.928768 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.928730 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:18.951819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:18.951762 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podStartSLOduration=0.951745091 podStartE2EDuration="951.745091ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:18.948424557 +0000 UTC m=+732.641405048" watchObservedRunningTime="2026-11-01 21:56:18.951745091 +0000 UTC m=+732.644725578" Nov 01 21:56:19.012513 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.012394 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp"] Nov 01 21:56:19.017205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.017183 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.022261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.021358 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:56:19.022261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.021540 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:56:19.022261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.021643 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:56:19.022261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.022204 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:56:19.025978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.025952 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp"] Nov 01 21:56:19.160977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.160938 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.161437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.161057 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdw2\" (UniqueName: \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.161437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.161210 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.262387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.262352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdw2\" (UniqueName: \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.262624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.262550 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.262725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.262679 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.263527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.263503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.265317 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.265288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.285106 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.285083 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdw2\" (UniqueName: \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\") pod \"maas-api-revoke-sub-0a9ffa0e-xwfsp\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.323845 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.323815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:19.346332 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.346308 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:19.446837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.446800 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:19.456466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.455563 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:19.456466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.455622 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:19.702210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.702167 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp"] Nov 01 21:56:19.717168 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:19.717096 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82f3d4c_373e_441f_90ae_f1fda46bf788.slice/crio-0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0 WatchSource:0}: Error finding container 0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0: Status 404 returned error can't find the container with id 0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0 Nov 01 21:56:19.932396 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.932358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" event={"ID":"f82f3d4c-373e-441f-90ae-f1fda46bf788","Type":"ContainerStarted","Data":"0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0"} Nov 01 21:56:19.964588 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:19.964512 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:20.126493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.125433 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:20.447157 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.447119 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:20.447552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.447181 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:20.943259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.943228 5036 generic.go:356] "Generic (PLEG): container finished" podID="f82f3d4c-373e-441f-90ae-f1fda46bf788" containerID="cef4e244f9fe6734de52a6b7e368d6d8b5b1381b0f4520591bf1a46f13acfc24" exitCode=0 Nov 01 21:56:20.944295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.944272 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" event={"ID":"f82f3d4c-373e-441f-90ae-f1fda46bf788","Type":"ContainerDied","Data":"cef4e244f9fe6734de52a6b7e368d6d8b5b1381b0f4520591bf1a46f13acfc24"} Nov 01 21:56:20.950794 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:20.950761 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:21.448467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:21.447836 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:21.448467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:21.447901 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:22.468474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:22.463545 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:22.468474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:22.463637 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:23.053474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.052480 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.053474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.052794 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.077521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.056157 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.077521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.059218 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:23.108468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.088943 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:23.171766 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.171727 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.177027 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.176998 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" event={"ID":"f82f3d4c-373e-441f-90ae-f1fda46bf788","Type":"ContainerDied","Data":"0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0"} Nov 01 21:56:23.177222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.177209 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0439ce817a0a015bf4b8d75d88fe2abf5dbb6b378910bee736f677c19a2fd3d0" Nov 01 21:56:23.177356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.177347 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp" Nov 01 21:56:23.217553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.216540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.295469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.289732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\" (UniqueName: \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\") pod \"f82f3d4c-373e-441f-90ae-f1fda46bf788\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " Nov 01 21:56:23.295469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.289838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\") pod \"f82f3d4c-373e-441f-90ae-f1fda46bf788\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " Nov 01 21:56:23.295469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.289885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\") pod \"f82f3d4c-373e-441f-90ae-f1fda46bf788\" (UID: \"f82f3d4c-373e-441f-90ae-f1fda46bf788\") " Nov 01 21:56:23.329155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.328536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:56:23.333466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.329600 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca" pod "f82f3d4c-373e-441f-90ae-f1fda46bf788" (UID: "f82f3d4c-373e-441f-90ae-f1fda46bf788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:23.365482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.345019 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca" pod "f82f3d4c-373e-441f-90ae-f1fda46bf788" (UID: "f82f3d4c-373e-441f-90ae-f1fda46bf788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:23.372476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.366258 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2" pod "f82f3d4c-373e-441f-90ae-f1fda46bf788" (UID: "f82f3d4c-373e-441f-90ae-f1fda46bf788"). InnerVolumeSpecName "kube-api-access-2pdw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:23.396465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.391119 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdw2\" (UniqueName: \"kubernetes.io/projected/f82f3d4c-373e-441f-90ae-f1fda46bf788-kube-api-access-2pdw2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:23.396465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.391149 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:23.396465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.391165 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82f3d4c-373e-441f-90ae-f1fda46bf788-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:23.472466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.458843 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:23.472466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.458902 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:23.553158 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:23.553118 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:24.458593 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:24.458546 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:24.458890 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:24.458858 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:24.597226 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:24.597140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:25.446853 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:25.446823 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 21:56:25.447025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:25.446875 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 21:56:26.024761 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:26.024726 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:26.738630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:26.738577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:27.388103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.388072 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:27.388679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.388652 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:27.389500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.389478 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 21:56:27.604558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.604519 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:27.647237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.647150 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8"] Nov 01 21:56:27.647772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.647750 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f82f3d4c-373e-441f-90ae-f1fda46bf788" containerName="revoke-keys" Nov 01 21:56:27.647772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.647768 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82f3d4c-373e-441f-90ae-f1fda46bf788" containerName="revoke-keys" Nov 01 21:56:27.647967 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.647873 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82f3d4c-373e-441f-90ae-f1fda46bf788" containerName="revoke-keys" Nov 01 21:56:27.651960 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.651938 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.658180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.658153 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:56:27.658931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.658909 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:56:27.659670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.659255 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:56:27.659835 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.659373 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:56:27.660471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.660419 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8"] Nov 01 21:56:27.726299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.726262 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.726465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.726377 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kr4r\" (UniqueName: \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.726465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.726457 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.827604 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.827568 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.827780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.827648 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kr4r\" (UniqueName: \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.827780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.827693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.828274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.828246 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.830111 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.830089 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.840371 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.840350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kr4r\" (UniqueName: \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\") pod \"maas-api-revoke-sub-02e546cd-hrvd8\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:27.975112 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:27.975049 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:28.270484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:28.266669 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8"] Nov 01 21:56:28.270484 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:28.266887 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9e2989_45b3_4887_bc38_43c5992e2513.slice/crio-a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263 WatchSource:0}: Error finding container a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263: Status 404 returned error can't find the container with id a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263 Nov 01 21:56:28.280042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:28.280012 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" event={"ID":"5b9e2989-45b3-4887-bc38-43c5992e2513","Type":"ContainerStarted","Data":"a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263"} Nov 01 21:56:28.328182 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:28.328151 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:29.141969 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:29.141934 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:29.288127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:29.288091 5036 generic.go:356] "Generic (PLEG): container finished" podID="5b9e2989-45b3-4887-bc38-43c5992e2513" containerID="06b6f1cf3e8f9d3b69b7a565aa2919330c7b0120a09d039e725a97e37b81a4ef" exitCode=0 Nov 01 21:56:29.288373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:29.288182 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" event={"ID":"5b9e2989-45b3-4887-bc38-43c5992e2513","Type":"ContainerDied","Data":"06b6f1cf3e8f9d3b69b7a565aa2919330c7b0120a09d039e725a97e37b81a4ef"} Nov 01 21:56:29.838860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:29.838821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:30.476955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.476921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:30.488936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.488905 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:30.572516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.559963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\" (UniqueName: \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\") pod \"5b9e2989-45b3-4887-bc38-43c5992e2513\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " Nov 01 21:56:30.572516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.560082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\") pod \"5b9e2989-45b3-4887-bc38-43c5992e2513\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " Nov 01 21:56:30.572516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.560163 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\") pod \"5b9e2989-45b3-4887-bc38-43c5992e2513\" (UID: \"5b9e2989-45b3-4887-bc38-43c5992e2513\") " Nov 01 21:56:30.572516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.571570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca" pod "5b9e2989-45b3-4887-bc38-43c5992e2513" (UID: "5b9e2989-45b3-4887-bc38-43c5992e2513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:30.605566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.591031 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r" pod "5b9e2989-45b3-4887-bc38-43c5992e2513" (UID: "5b9e2989-45b3-4887-bc38-43c5992e2513"). InnerVolumeSpecName "kube-api-access-4kr4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:30.621920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.611942 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca" pod "5b9e2989-45b3-4887-bc38-43c5992e2513" (UID: "5b9e2989-45b3-4887-bc38-43c5992e2513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:30.666332 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.666294 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:30.666332 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.666333 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9e2989-45b3-4887-bc38-43c5992e2513-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:30.666587 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.666348 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kr4r\" (UniqueName: \"kubernetes.io/projected/5b9e2989-45b3-4887-bc38-43c5992e2513-kube-api-access-4kr4r\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:30.789040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.788996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:30.825555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.821845 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z"] Nov 01 21:56:30.825555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.822793 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5b9e2989-45b3-4887-bc38-43c5992e2513" containerName="revoke-keys" Nov 01 21:56:30.825555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.822815 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9e2989-45b3-4887-bc38-43c5992e2513" containerName="revoke-keys" Nov 01 21:56:30.825555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.822964 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9e2989-45b3-4887-bc38-43c5992e2513" containerName="revoke-keys" Nov 01 21:56:30.827221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.827197 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.833524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.833499 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z"] Nov 01 21:56:30.868269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.868237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdb2\" (UniqueName: \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.868408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.868311 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.868508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.868476 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.969566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.969529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.969748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.969677 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdb2\" (UniqueName: \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.969748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.969732 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.970228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.970204 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.972071 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.972049 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:30.982303 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:30.982281 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdb2\" (UniqueName: \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\") pod \"maas-api-revoke-sub-5a7dce8f-k9w4z\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:31.148091 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.148010 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:31.307497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.305155 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" Nov 01 21:56:31.307497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.305181 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8" event={"ID":"5b9e2989-45b3-4887-bc38-43c5992e2513","Type":"ContainerDied","Data":"a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263"} Nov 01 21:56:31.307497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.305231 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a155ad4a93859509f78084ef9f4e1a61d31e8fde773b70fbccaa2e3941dc4263" Nov 01 21:56:31.434808 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.434781 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z"] Nov 01 21:56:31.435566 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:31.435544 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f1b363_b6d8_40a9_9369_aa76e109e994.slice/crio-f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1 WatchSource:0}: Error finding container f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1: Status 404 returned error can't find the container with id f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1 Nov 01 21:56:31.521422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:31.521383 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:32.314757 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:32.314724 5036 generic.go:356] "Generic (PLEG): container finished" podID="d4f1b363-b6d8-40a9-9369-aa76e109e994" containerID="857b1525d88658a750a82e7c0324cd93e9cc9636eb85750878b1faab199d7676" exitCode=0 Nov 01 21:56:32.314866 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:32.314796 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" event={"ID":"d4f1b363-b6d8-40a9-9369-aa76e109e994","Type":"ContainerDied","Data":"857b1525d88658a750a82e7c0324cd93e9cc9636eb85750878b1faab199d7676"} Nov 01 21:56:32.314866 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:32.314848 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" event={"ID":"d4f1b363-b6d8-40a9-9369-aa76e109e994","Type":"ContainerStarted","Data":"f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1"} Nov 01 21:56:32.418212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:32.418176 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:33.128893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.128840 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:33.145235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.145195 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 21:56:33.149966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.149939 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.162063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.161980 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 21:56:33.196086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.196058 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.196791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.196686 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.196920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.196874 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.196993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.196968 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.197061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.197042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.197193 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.197170 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkr4\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.197414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.197385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.197526 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.197478 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.197658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.197519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298726 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298770 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298845 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkr4\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298880 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.298937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298953 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.298997 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299035 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299223 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299378 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299285 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299550 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.299812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.299785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.301484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.301436 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.303818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.303791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.311181 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.311140 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.314070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.314043 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkr4\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.478668 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.478641 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:33.478944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.478914 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:33.603200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.603169 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\") pod \"d4f1b363-b6d8-40a9-9369-aa76e109e994\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " Nov 01 21:56:33.603430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.603217 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\" (UniqueName: \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\") pod \"d4f1b363-b6d8-40a9-9369-aa76e109e994\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " Nov 01 21:56:33.603430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.603352 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\") pod \"d4f1b363-b6d8-40a9-9369-aa76e109e994\" (UID: \"d4f1b363-b6d8-40a9-9369-aa76e109e994\") " Nov 01 21:56:33.603698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.603677 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca" pod "d4f1b363-b6d8-40a9-9369-aa76e109e994" (UID: "d4f1b363-b6d8-40a9-9369-aa76e109e994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:33.603906 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.603889 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:33.606061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.606024 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2" pod "d4f1b363-b6d8-40a9-9369-aa76e109e994" (UID: "d4f1b363-b6d8-40a9-9369-aa76e109e994"). InnerVolumeSpecName "kube-api-access-9cdb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:33.606061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.606054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca" pod "d4f1b363-b6d8-40a9-9369-aa76e109e994" (UID: "d4f1b363-b6d8-40a9-9369-aa76e109e994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:33.704771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.704699 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdb2\" (UniqueName: \"kubernetes.io/projected/d4f1b363-b6d8-40a9-9369-aa76e109e994-kube-api-access-9cdb2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:33.704771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.704726 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f1b363-b6d8-40a9-9369-aa76e109e994-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:33.773482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.773055 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 21:56:33.773482 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:33.773315 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b17b3d_7159_4bd5_b22d_d3eecaca151c.slice/crio-1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0 WatchSource:0}: Error finding container 1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0: Status 404 returned error can't find the container with id 1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0 Nov 01 21:56:33.775955 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.775925 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:33.776043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.775991 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:33.776043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.776023 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:56:33.831575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:33.831537 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:34.333359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.333322 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" event={"ID":"88b17b3d-7159-4bd5-b22d-d3eecaca151c","Type":"ContainerStarted","Data":"c32b88e92c630ccf9335c658e7f2308464fde0830b6142be5646ae90b5b89a58"} Nov 01 21:56:34.333789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.333377 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" event={"ID":"88b17b3d-7159-4bd5-b22d-d3eecaca151c","Type":"ContainerStarted","Data":"1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0"} Nov 01 21:56:34.334966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.334945 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" Nov 01 21:56:34.335052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.334963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z" event={"ID":"d4f1b363-b6d8-40a9-9369-aa76e109e994","Type":"ContainerDied","Data":"f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1"} Nov 01 21:56:34.335052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.335001 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c8197e79e41de1e881b153a6bcdac1f64bca92f2498c0d8996f542cdaec6c1" Nov 01 21:56:34.357965 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.357916 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podStartSLOduration=1.357903331 podStartE2EDuration="1.357903331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:34.356039857 +0000 UTC m=+748.049020368" watchObservedRunningTime="2026-11-01 21:56:34.357903331 +0000 UTC m=+748.050883817" Nov 01 21:56:34.485474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.481571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:34.485474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.483556 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:34.485474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.483622 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:34.531419 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:34.531379 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:35.259224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:35.259180 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:35.483464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:35.480805 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:35.483464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:35.480862 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:35.973905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:35.973868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:36.110468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:36.109345 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:36.497474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:36.480809 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:36.497474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:36.480867 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:36.861672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:36.861633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:37.322114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.322077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:56:37.480468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.479806 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:37.480468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.479878 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:37.661046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.660960 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:37.959429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.959354 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn"] Nov 01 21:56:37.960073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.960053 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d4f1b363-b6d8-40a9-9369-aa76e109e994" containerName="revoke-keys" Nov 01 21:56:37.960073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.960072 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f1b363-b6d8-40a9-9369-aa76e109e994" containerName="revoke-keys" Nov 01 21:56:37.960219 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.960189 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f1b363-b6d8-40a9-9369-aa76e109e994" containerName="revoke-keys" Nov 01 21:56:37.964011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.963993 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:37.969021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.968995 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:56:37.969148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.969068 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:56:37.969243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.969145 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:56:37.969308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.969251 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:56:37.970342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:37.970320 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn"] Nov 01 21:56:38.199537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.199072 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5xr\" (UniqueName: \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.199537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.199299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.199537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.199334 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.305047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.305015 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5xr\" (UniqueName: \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.305792 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.305770 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.305966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.305949 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.306880 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.306856 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.311895 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.311864 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.318102 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.318084 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5xr\" (UniqueName: \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\") pod \"maas-api-revoke-sub-e60c217e-bpwdn\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.318762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.318746 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:38.483467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.482800 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:38.483467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.482876 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:38.659827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.659793 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn"] Nov 01 21:56:38.777465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:38.774318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:39.470470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:39.468536 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" event={"ID":"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4","Type":"ContainerStarted","Data":"636a9602f9686bcc2e3e5273a57dde439492492457f2e8ec583c3fd1fcc21570"} Nov 01 21:56:39.470470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:39.468660 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" event={"ID":"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4","Type":"ContainerStarted","Data":"e00b7bb640e461572ab06d3b2973edf254a311d2b25c902a709bcad1ed757255"} Nov 01 21:56:39.485964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:39.485920 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:39.516648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:39.516603 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:39.530467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:39.528893 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" podStartSLOduration=2.528873874 podStartE2EDuration="2.528873874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:39.522977722 +0000 UTC m=+753.215958209" watchObservedRunningTime="2026-11-01 21:56:39.528873874 +0000 UTC m=+753.221854361" Nov 01 21:56:40.471092 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:40.471054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:40.481800 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:40.481769 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 21:56:40.482055 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:40.482028 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 21:56:40.493334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:40.493303 5036 generic.go:356] "Generic (PLEG): container finished" podID="43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" containerID="636a9602f9686bcc2e3e5273a57dde439492492457f2e8ec583c3fd1fcc21570" exitCode=0 Nov 01 21:56:40.493689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:40.493667 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" event={"ID":"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4","Type":"ContainerDied","Data":"636a9602f9686bcc2e3e5273a57dde439492492457f2e8ec583c3fd1fcc21570"} Nov 01 21:56:41.251516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.251413 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:41.753280 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.753250 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:41.817464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.816790 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\" (UniqueName: \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\") pod \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " Nov 01 21:56:41.817464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.816863 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\") pod \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " Nov 01 21:56:41.817464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.816932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\") pod \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\" (UID: \"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4\") " Nov 01 21:56:41.828318 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.828281 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca" pod "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" (UID: "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:41.845154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.845124 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca" pod "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" (UID: "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:41.845637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.845610 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr" pod "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" (UID: "43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4"). InnerVolumeSpecName "kube-api-access-tc5xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:41.926058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.926025 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:41.926294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.926278 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5xr\" (UniqueName: \"kubernetes.io/projected/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-kube-api-access-tc5xr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:41.926403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.926391 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:41.946873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:41.946839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:42.022871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.021516 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:42.022871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.022806 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:42.024774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.024754 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 21:56:42.516858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.516832 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" Nov 01 21:56:42.517046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.516831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn" event={"ID":"43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4","Type":"ContainerDied","Data":"e00b7bb640e461572ab06d3b2973edf254a311d2b25c902a709bcad1ed757255"} Nov 01 21:56:42.517046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.516970 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00b7bb640e461572ab06d3b2973edf254a311d2b25c902a709bcad1ed757255" Nov 01 21:56:42.642611 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:42.642572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:43.426380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.426347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:43.474660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.474625 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 21:56:43.475251 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.475223 5036 cpu_manager.go:414] "containerMap: removing container" podUID="43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" containerName="revoke-keys" Nov 01 21:56:43.475251 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.475245 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" containerName="revoke-keys" Nov 01 21:56:43.475433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.475342 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" containerName="revoke-keys" Nov 01 21:56:43.479080 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.479057 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.490370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.490200 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 21:56:43.493025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.493000 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"distinct-2-w0-1f3a6b-kserve-self-signed-certs\"" Nov 01 21:56:43.544056 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfr2\" (UniqueName: \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.544227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.544227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544165 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.544366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544336 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.544882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544849 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.544882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:43.545058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.544954 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.646706 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.646786 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.646849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.646961 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfr2\" (UniqueName: \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.647005 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.647465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.647032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.651011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.650427 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.651529 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.651497 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.663430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.652170 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.663430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.662791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.664356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.664333 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.667520 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.667496 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfr2\" (UniqueName: \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\") pod \"distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:43.796954 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:43.796920 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:44.087291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.087266 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 21:56:44.089653 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:44.089626 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c044010_8ee6_4df8_a258_6b71c1e97e80.slice/crio-4509ef0b6d104e3f22d990d03a9d7624703a4293b92ba608351921642f4056e5 WatchSource:0}: Error finding container 4509ef0b6d104e3f22d990d03a9d7624703a4293b92ba608351921642f4056e5: Status 404 returned error can't find the container with id 4509ef0b6d104e3f22d990d03a9d7624703a4293b92ba608351921642f4056e5 Nov 01 21:56:44.128022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.127983 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:44.538749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.538715 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" event={"ID":"2c044010-8ee6-4df8-a258-6b71c1e97e80","Type":"ContainerStarted","Data":"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa"} Nov 01 21:56:44.539127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.538766 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" event={"ID":"2c044010-8ee6-4df8-a258-6b71c1e97e80","Type":"ContainerStarted","Data":"4509ef0b6d104e3f22d990d03a9d7624703a4293b92ba608351921642f4056e5"} Nov 01 21:56:44.560554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.560505 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podStartSLOduration=1.560492928 podStartE2EDuration="1.560492928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:44.558476369 +0000 UTC m=+758.251456855" watchObservedRunningTime="2026-11-01 21:56:44.560492928 +0000 UTC m=+758.253473414" Nov 01 21:56:44.797982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.797907 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:44.937337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:44.937294 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:45.736721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:45.736692 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:46.513210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:46.513176 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:47.230804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:47.230755 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:47.941922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:47.941890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:48.751411 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:48.751365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:48.859012 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:56:48.858974 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:56:49.416082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:49.416041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:49.862517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:49.862486 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:50.668374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:50.668338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:50.894711 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:50.894676 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:51.658406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:51.658365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:52.420024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.419985 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:52.808618 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.808583 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 21:56:52.815461 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.815422 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.816900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.816864 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 21:56:52.822569 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.822532 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"distinct-w2-f033b1-kserve-self-signed-certs\"" Nov 01 21:56:52.878092 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.878057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:52.968307 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.968267 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.968592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.968572 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.968790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.968769 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.968935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.968919 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92824\" (UniqueName: \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.969040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.969026 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:52.969140 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:52.969127 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.069974 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070026 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92824\" (UniqueName: \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070053 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070212 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070438 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070625 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.070736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.070721 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.072543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.072518 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.073073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.073054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.082718 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.082690 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92824\" (UniqueName: \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\") pod \"distinct-w2-f033b1-kserve-f7774fbf8-qgzz2\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.135028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.134840 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:53.404722 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.404630 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:53.436314 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:56:53.436282 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1edb3f3e_2731_4d2e_ab4e_4c5c83703602.slice/crio-5623d6b44211a2e3e5adaf629959367418256e5345086c7e4b4e70580473e1e0 WatchSource:0}: Error finding container 5623d6b44211a2e3e5adaf629959367418256e5345086c7e4b4e70580473e1e0: Status 404 returned error can't find the container with id 5623d6b44211a2e3e5adaf629959367418256e5345086c7e4b4e70580473e1e0 Nov 01 21:56:53.436783 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.436760 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 21:56:53.620134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.620095 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" event={"ID":"1edb3f3e-2731-4d2e-ab4e-4c5c83703602","Type":"ContainerStarted","Data":"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b"} Nov 01 21:56:53.620593 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.620144 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" event={"ID":"1edb3f3e-2731-4d2e-ab4e-4c5c83703602","Type":"ContainerStarted","Data":"5623d6b44211a2e3e5adaf629959367418256e5345086c7e4b4e70580473e1e0"} Nov 01 21:56:53.639865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.639813 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podStartSLOduration=1.6397966259999999 podStartE2EDuration="1.639796626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:53.639311168 +0000 UTC m=+767.332291655" watchObservedRunningTime="2026-11-01 21:56:53.639796626 +0000 UTC m=+767.332777117" Nov 01 21:56:53.797544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.797509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:53.819370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:53.819336 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:54.135313 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:54.135229 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:56:54.242600 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:54.242563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:54.639732 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:54.639703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:56:54.942194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:54.942106 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:55.741999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:55.741959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:56.349464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:56.345293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:56.735527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:56.735405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:57.788491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:57.788430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:58.636497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:58.636397 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:58.772343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:58.772301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:59.342999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:56:59.342959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:00.038430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:00.038395 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:00.843725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:00.843691 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:01.523387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:01.523348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:02.214497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.214435 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:02.691776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.691744 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 21:57:02.696762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.696734 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 21:57:02.696934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.696877 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.702874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.702841 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.706970 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.706944 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0def36-openshift-default-dockercfg-wfkj8\"" Nov 01 21:57:02.712104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.712084 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f3a6b-openshift-default-dockercfg-nlg9b\"" Nov 01 21:57:02.714127 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.714108 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 21:57:02.718436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.718411 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 21:57:02.804536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.804688 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804615 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.804688 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804636 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.804688 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.804827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804799 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzr2t\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.804881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804864 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.804925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.804969 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.804954 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805059 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805146 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805268 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.805380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805366 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805458 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805403 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.805525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vpd\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.805525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.805592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805527 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.805592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.805587 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.906781 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.906749 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.906781 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.906785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.906807 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vpd\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.907024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.906829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.906952 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907005 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907057 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907150 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.907235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907178 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907210 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907218 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.907639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzr2t\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.907639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907556 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907903 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907922 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.907951 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908004 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908085 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908237 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908281 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908649 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.908839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.909840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.909305 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.910215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.910170 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.910795 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.910770 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.911568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.911547 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.911704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.911685 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.919860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.919834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vpd\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.920048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.920030 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:02.920367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.920347 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzr2t\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.920405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.920378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:02.954617 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:02.954527 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:03.012754 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.012713 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 21:57:03.019034 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.019015 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.025887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.025860 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:03.033517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.032910 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 21:57:03.041292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.041271 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:03.112839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.112807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjdt\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.112876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.112959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113112 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.113202 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.113192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.114256 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.114232 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 21:57:03.131506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.130723 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 21:57:03.131506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.130861 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.138010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.137882 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:03.163068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.162429 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216113 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9m9\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216175 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjdt\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216224 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216324 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216363 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216415 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216460 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216550 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216577 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216601 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216676 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216700 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216755 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.216786 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.217545 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.218289 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.225282 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.226791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.225963 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.228830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.228532 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.235521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.232860 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.235521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.233536 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.239240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.236826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjdt\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.240863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.240679 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.318678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318640 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9m9\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.318856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318789 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.318856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.318856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318889 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.318972 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319025 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319086 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319543 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319560 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319631 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.319962 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.319940 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.321289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.321267 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.321592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.321573 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.336944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.336919 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 21:57:03.337280 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:03.337255 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-cd3a95ac2ac95f8c7bd097ccb63a15256c9d25a1547f17f6728bdfac7e197af9 WatchSource:0}: Error finding container cd3a95ac2ac95f8c7bd097ccb63a15256c9d25a1547f17f6728bdfac7e197af9: Status 404 returned error can't find the container with id cd3a95ac2ac95f8c7bd097ccb63a15256c9d25a1547f17f6728bdfac7e197af9 Nov 01 21:57:03.338258 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.338212 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9m9\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.338430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.338410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.340159 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.340132 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.340227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.340190 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.340227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.340220 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.341551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.341535 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:03.348754 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.348017 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 21:57:03.355299 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:03.355272 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2493eb28_61df_4874_a82b_d2ce69e3f1bd.slice/crio-7efa3f4ac5be6717aa54950f367ad6aec493f33cdf8457d4cda86a47547aa829 WatchSource:0}: Error finding container 7efa3f4ac5be6717aa54950f367ad6aec493f33cdf8457d4cda86a47547aa829: Status 404 returned error can't find the container with id 7efa3f4ac5be6717aa54950f367ad6aec493f33cdf8457d4cda86a47547aa829 Nov 01 21:57:03.362578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.362549 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.362676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.362623 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.362676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.362667 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.453428 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.452993 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:03.656608 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:03.656577 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff83270d_a1d8_4357_b365_a5eac7061e66.slice/crio-1fb15ae7420af11857595d238c14e7bd15670688cd9095b47fe3cb0f4152b459 WatchSource:0}: Error finding container 1fb15ae7420af11857595d238c14e7bd15670688cd9095b47fe3cb0f4152b459: Status 404 returned error can't find the container with id 1fb15ae7420af11857595d238c14e7bd15670688cd9095b47fe3cb0f4152b459 Nov 01 21:57:03.658500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.658476 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 21:57:03.659093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.659061 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.659175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.659125 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.659175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.659152 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.732808 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.732768 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" event={"ID":"2493eb28-61df-4874-a82b-d2ce69e3f1bd","Type":"ContainerStarted","Data":"35f7ff00745ee5ce8ab52c0ffbe57d9e3a429e202d8599e0befad084612e09ea"} Nov 01 21:57:03.732926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.732833 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" event={"ID":"2493eb28-61df-4874-a82b-d2ce69e3f1bd","Type":"ContainerStarted","Data":"7efa3f4ac5be6717aa54950f367ad6aec493f33cdf8457d4cda86a47547aa829"} Nov 01 21:57:03.737292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.737228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" event={"ID":"ff83270d-a1d8-4357-b365-a5eac7061e66","Type":"ContainerStarted","Data":"1fb15ae7420af11857595d238c14e7bd15670688cd9095b47fe3cb0f4152b459"} Nov 01 21:57:03.739390 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.739365 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" event={"ID":"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc","Type":"ContainerStarted","Data":"a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb"} Nov 01 21:57:03.739508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.739414 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" event={"ID":"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc","Type":"ContainerStarted","Data":"cd3a95ac2ac95f8c7bd097ccb63a15256c9d25a1547f17f6728bdfac7e197af9"} Nov 01 21:57:03.755686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.755656 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:03.799707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.798655 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podStartSLOduration=1.798636627 podStartE2EDuration="1.798636627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:03.773117852 +0000 UTC m=+777.466098337" watchObservedRunningTime="2026-11-01 21:57:03.798636627 +0000 UTC m=+777.491617116" Nov 01 21:57:03.810610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.810224 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.810610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.810299 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.810610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.810344 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:03.822623 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.822588 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 21:57:03.843339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.843283 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podStartSLOduration=1.843266329 podStartE2EDuration="1.843266329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:03.837225888 +0000 UTC m=+777.530206390" watchObservedRunningTime="2026-11-01 21:57:03.843266329 +0000 UTC m=+777.536246816" Nov 01 21:57:03.845664 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:03.845529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:04.035517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.035486 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:04.044749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.044659 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:04.050903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.050564 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:04.050903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.050624 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:04.051252 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.051232 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:04.051414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.051394 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:04.690080 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.690040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:04.807807 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.807767 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" event={"ID":"a39f88cf-9031-4d5e-86fe-233ae699a781","Type":"ContainerStarted","Data":"cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b"} Nov 01 21:57:04.813654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.813618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" event={"ID":"a39f88cf-9031-4d5e-86fe-233ae699a781","Type":"ContainerStarted","Data":"44bae5cb506dce92156ee1976d165f38ba493e62a2027d3c1f388fdaaefcd1e5"} Nov 01 21:57:04.837508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.834235 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" event={"ID":"ff83270d-a1d8-4357-b365-a5eac7061e66","Type":"ContainerStarted","Data":"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada"} Nov 01 21:57:04.874469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.874049 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podStartSLOduration=1.874027377 podStartE2EDuration="1.874027377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.858089815 +0000 UTC m=+778.551070314" watchObservedRunningTime="2026-11-01 21:57:04.874027377 +0000 UTC m=+778.567007927" Nov 01 21:57:04.967524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:04.964862 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podStartSLOduration=2.964842424 podStartE2EDuration="2.964842424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.950825088 +0000 UTC m=+778.643805577" watchObservedRunningTime="2026-11-01 21:57:04.964842424 +0000 UTC m=+778.657822912" Nov 01 21:57:05.029648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.029612 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.030075 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.030047 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:05.044413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.044376 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.044705 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.044672 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:05.346576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.346496 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:05.346910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.346885 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.347073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.347046 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:05.458305 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.458266 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:05.458663 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.458639 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.458829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.458805 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:05.904496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.901165 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:05.935655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.935610 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:05.936629 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.936595 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:05.956259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.956216 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:57:05.995997 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:05.995953 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.027467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.026866 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.027467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.026933 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:06.047468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.047121 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.047468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.047186 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:06.134469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.134088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.153474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.146701 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:06.294469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.292550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.350470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.348771 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.350470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.348849 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:06.505476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.491544 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.505476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.491602 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:06.558526 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.558331 5036 scope.go:122] "RemoveContainer" containerID="411b942b017430c2ee9f5d699fd139cde477bddf84d6714d0d68acce09b001f7" Nov 01 21:57:06.632927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.631839 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:57:06.743736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.743689 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:06.937671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.937560 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.940691 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.939242 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:06.989680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.989638 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.990477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:06.990437 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:57:07.003046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.003009 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:07.005975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.005957 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:07.070852 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.070820 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.071094 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.071070 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:07.101731 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.090458 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.123784 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.123696 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:07.151655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.150529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.151655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.150973 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:07.265703 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.265593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.313464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.312826 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:07.362467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.357782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:07.446475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.444595 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.446475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.444658 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:07.562510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.561567 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.562510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.561626 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:07.603468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.591327 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:07.603468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.591395 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:07.633554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.633513 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.644692 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.634251 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:57:07.721571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.721543 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:07.741253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.729845 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:57:07.793634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.761850 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.793634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.787346 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:07.816086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.815621 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:57:07.971673 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.969955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.971673 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:07.970312 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:08.043903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.037805 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.043903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.037866 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:08.045680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.045095 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.045680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.045143 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:08.268070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.267990 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:08.346342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.346308 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.365696 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.365657 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:08.461670 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.461631 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.462043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.462012 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:08.541305 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.541265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:08.615469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.603576 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:08.795584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:08.791771 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.030912 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.030873 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.031517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.031469 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:09.041477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.041417 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.041700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.041677 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:09.269302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.268282 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:57:09.315115 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.313734 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:09.355070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.355035 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.355316 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.355290 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:09.456468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.454479 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.456468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.454535 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:09.524521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.510494 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 21:57:09.555462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.545804 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:09.571740 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.571709 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.596532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.594364 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"distinct-2-w3-ff3e70-kserve-self-signed-certs\"" Nov 01 21:57:09.604577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.604540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.617479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.616017 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672271 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672511 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672595 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.672712 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52sp\" (UniqueName: \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.732535 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.755255 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.747715 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.790531 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.774337 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.799599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.799561 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.824430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.824544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.824691 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52sp\" (UniqueName: \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.824752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.829570 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.778002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.830345 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.800135 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.864590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.832844 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:09.888858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:09.888825 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 21:57:10.042620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.042530 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.055892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.055775 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:10.056553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.043162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52sp\" (UniqueName: \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\") pod \"distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:10.056553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.044282 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.056845 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.056783 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:10.196100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.196065 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:10.200858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.200816 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:10.404516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.401024 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.404516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.401100 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:10.404516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.403111 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:10.463839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.463810 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.464042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:10.464015 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:11.031573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.028218 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.031573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.028282 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:11.047476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.046208 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.047476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.046272 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:11.275926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.246762 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.359486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.346519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.359486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.348045 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.359486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.348086 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:11.412980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.409977 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:11.412980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.412220 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.480876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.480842 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.481910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.481881 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:11.691917 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.682773 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.725460 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.725404 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.739480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.727088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:11.757821 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.751837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.786213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.770572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:11.828484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.819547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.866498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.863264 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 21:57:11.942681 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:11.942589 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.047492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.043341 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.047492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.043412 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:12.047492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.044752 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.047492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.044800 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:12.253530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.247022 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.253530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.248577 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:57:12.351887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.351836 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.352892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.352869 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:12.412552 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.412504 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.412853 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.412807 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:12.433905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.418608 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.433905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.418895 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:12.433905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.424487 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.433905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.425531 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:12.463063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.463026 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.463314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.463282 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:12.659991 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.659958 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:12.673528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.673493 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:12.688588 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.683643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:57:12.689352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.689310 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:12.718553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.701632 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" event={"ID":"23bb7067-f212-41c7-b4bb-b18d94c1953f","Type":"ContainerStarted","Data":"0aa70dd287eb70113cda38fdb676fad7b61c5bf36477a409ca5c390ca91b93b3"} Nov 01 21:57:12.732867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.732839 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:12.735524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.735496 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:12.736209 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.736191 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:12.754798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.754616 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.755008 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.754982 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:57:12.827589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.827545 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:12.840406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.840378 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:57:12.977564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:12.951962 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:13.268346 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.268254 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:57:13.270654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.270628 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:13.270825 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.270806 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:13.270969 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.270925 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:13.271179 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.271155 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:13.340500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.292728 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:57:13.344885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.344773 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:57:13.344885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.344829 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 21:57:13.383599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.383294 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:57:13.427578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.427543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:13.450883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.450863 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:13.578874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.578484 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:57:13.578874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.578715 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:57:13.578874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.578760 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 21:57:13.625069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.618211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:13.707186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.707141 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:13.779178 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.779141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" event={"ID":"23bb7067-f212-41c7-b4bb-b18d94c1953f","Type":"ContainerStarted","Data":"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43"} Nov 01 21:57:13.821535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.820197 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:13.845837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:13.845731 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podStartSLOduration=4.845712033 podStartE2EDuration="4.845712033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:13.808663055 +0000 UTC m=+787.501643543" watchObservedRunningTime="2026-11-01 21:57:13.845712033 +0000 UTC m=+787.538692520" Nov 01 21:57:14.026865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.026818 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 21:57:14.027151 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.027116 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 21:57:14.048674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.043102 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 21:57:14.048674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.043155 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 21:57:14.203337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.203234 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:14.336804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.336604 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:14.520532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.518126 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:14.520532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.519792 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:14.521332 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.521313 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 21:57:14.578123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.578100 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:14.579765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.579743 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:14.598532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:14.598499 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 21:57:15.037671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.036553 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:15.037671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.036976 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:15.037671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.037648 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 21:57:15.048360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.048329 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:15.344390 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.344305 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:15.344832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.344811 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:15.345926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.345904 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 21:57:15.727219 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:15.727128 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:16.425586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:16.425522 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:17.220910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.220877 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:17.787291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.787258 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 21:57:17.792700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.792675 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.808470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.808430 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 21:57:17.909976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.909934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910139 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.909985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910139 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910077 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtfn\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910223 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910270 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910296 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910460 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:17.910543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:17.910529 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011365 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011408 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011464 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011629 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011687 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011904 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011714 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtfn\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011904 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.011904 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.011819 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.012314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.012287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.012675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.012646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.012824 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.012772 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.013072 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.013050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.013285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.013241 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.015548 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.015522 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.020235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.020208 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.023334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.023314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:18.023974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.023948 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.025302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.025278 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtfn\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.055076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.054997 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g"] Nov 01 21:57:18.063500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.061713 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g"] Nov 01 21:57:18.063500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.061822 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.066934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.066906 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:57:18.067070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.066921 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:57:18.079840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.079816 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:57:18.093076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.093048 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:57:18.121388 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.121343 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4cb4\" (UniqueName: \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.121796 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.121769 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.122100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.122022 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.122593 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.122558 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:18.172418 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.171676 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 21:57:18.182514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.182475 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.195536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.194670 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 21:57:18.197343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.197107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:18.234050 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234013 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234078 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234109 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234175 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4cb4\" (UniqueName: \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.234535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234343 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234475 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.234535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234503 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8tp\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.234774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.234619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.240690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.240631 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.243560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.243539 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.254494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.252537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4cb4\" (UniqueName: \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\") pod \"maas-api-revoke-sub-d23db5a6-x8w6g\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.335775 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335498 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.335775 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335582 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8tp\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.335775 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335714 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.335775 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335787 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335856 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.335954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.336000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.336679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.336652 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.337175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.337155 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.337315 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.337291 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.337919 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.337897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.338035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.338009 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.340618 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.340595 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.341150 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.341110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.348872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.348849 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.349198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.349181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8tp\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.380302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.380276 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:18.543820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.543787 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 21:57:18.544051 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.544019 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.544128 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.544097 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.544187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.544142 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.553333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.553313 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:18.691819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.688602 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g"] Nov 01 21:57:18.693210 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:18.693167 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ed995b_5678_4339_83a0_0c1e261995f7.slice/crio-7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8 WatchSource:0}: Error finding container 7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8: Status 404 returned error can't find the container with id 7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8 Nov 01 21:57:18.845573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.845479 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" event={"ID":"a3ed995b-5678-4339-83a0-0c1e261995f7","Type":"ContainerStarted","Data":"7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8"} Nov 01 21:57:18.847619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.847596 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" event={"ID":"97fc8b6a-a114-49d3-9e5d-1e4bc121f200","Type":"ContainerStarted","Data":"051d56bb053bd96e7332364cbcb014bb4ae68606c52336dfdcc02ceb88b8c12a"} Nov 01 21:57:18.847776 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.847641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" event={"ID":"97fc8b6a-a114-49d3-9e5d-1e4bc121f200","Type":"ContainerStarted","Data":"37d6566bbfa7b78a2616970fdffffc0c6913ce5ec4a6d251bc51bd06cfb289f1"} Nov 01 21:57:18.862904 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:57:18.862875 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:57:18.863082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.863041 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 21:57:18.863884 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:18.863861 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb90432_17a2_41b1_9fbc_926a535dfc14.slice/crio-5a3637f7a246b882c3b80cdd99be0346023e29a4b770e6ff2f8d4b9f9cce62bb WatchSource:0}: Error finding container 5a3637f7a246b882c3b80cdd99be0346023e29a4b770e6ff2f8d4b9f9cce62bb: Status 404 returned error can't find the container with id 5a3637f7a246b882c3b80cdd99be0346023e29a4b770e6ff2f8d4b9f9cce62bb Nov 01 21:57:18.866438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.866404 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.866543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.866506 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.866582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.866555 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:18.877813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.877764 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podStartSLOduration=1.877735041 podStartE2EDuration="1.877735041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:18.875864627 +0000 UTC m=+792.568845114" watchObservedRunningTime="2026-11-01 21:57:18.877735041 +0000 UTC m=+792.570715528" Nov 01 21:57:18.942357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:18.942318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:19.123636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.123545 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:19.125184 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.125153 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:19.125348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.125212 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:19.296222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.296188 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8"] Nov 01 21:57:19.303942 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.303912 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.309708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.309679 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8"] Nov 01 21:57:19.364858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.364778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7424m\" (UniqueName: \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.365244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.365220 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.365530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.365499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.466822 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.466745 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.466960 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.466857 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7424m\" (UniqueName: \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.467010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.466985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.467938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.467911 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.469705 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.469680 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.504584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.492272 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7424m\" (UniqueName: \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\") pod \"maas-api-revoke-sub-4afbe64c-62md8\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.632244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.632194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:19.635167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.635140 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:19.898204 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.897678 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" event={"ID":"cfb90432-17a2-41b1-9fbc-926a535dfc14","Type":"ContainerStarted","Data":"2d09c076f2867e094975cd3b50ba04e35972a9cd0ca1fad5584dccc31c61608f"} Nov 01 21:57:19.898204 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.897753 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" event={"ID":"cfb90432-17a2-41b1-9fbc-926a535dfc14","Type":"ContainerStarted","Data":"5a3637f7a246b882c3b80cdd99be0346023e29a4b770e6ff2f8d4b9f9cce62bb"} Nov 01 21:57:19.905479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.900988 5036 generic.go:356] "Generic (PLEG): container finished" podID="a3ed995b-5678-4339-83a0-0c1e261995f7" containerID="255e394d0c87acb0223cbd2567d0bbb528ff6de0419786d98a67113c02455327" exitCode=0 Nov 01 21:57:19.915472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.907157 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" event={"ID":"a3ed995b-5678-4339-83a0-0c1e261995f7","Type":"ContainerDied","Data":"255e394d0c87acb0223cbd2567d0bbb528ff6de0419786d98a67113c02455327"} Nov 01 21:57:19.969272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:19.969016 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podStartSLOduration=1.96899646 podStartE2EDuration="1.96899646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:19.963157539 +0000 UTC m=+793.656138026" watchObservedRunningTime="2026-11-01 21:57:19.96899646 +0000 UTC m=+793.661976947" Nov 01 21:57:20.140565 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.119959 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8"] Nov 01 21:57:20.171067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.165281 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:20.171067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.165332 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:20.206470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.204620 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:20.235403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.235367 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:20.455829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.455636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:20.560467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.559827 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:20.565478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.561635 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:20.565478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.561692 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:20.830516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.830211 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:57:20.950505 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.950458 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" event={"ID":"2fd51b98-183c-42be-a6d1-643ec4454d24","Type":"ContainerStarted","Data":"c822f41f19b1febb68c7e62630893c5e1eb9386e9df088c00b15d95f87604c8f"} Nov 01 21:57:20.950914 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.950519 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" event={"ID":"2fd51b98-183c-42be-a6d1-643ec4454d24","Type":"ContainerStarted","Data":"68a78f151911afe2e1498f653be94868cdfd6e8df333ed6c518070f550f129d0"} Nov 01 21:57:20.964149 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.964123 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:20.972234 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:20.972174 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" podStartSLOduration=1.972157555 podStartE2EDuration="1.972157555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:20.97126328 +0000 UTC m=+794.664243769" watchObservedRunningTime="2026-11-01 21:57:20.972157555 +0000 UTC m=+794.665138044" Nov 01 21:57:21.137187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.134717 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:21.137187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.134782 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:21.376253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.376220 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:21.490546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.490516 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:21.555909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.555877 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:21.556161 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.556120 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:21.572146 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.572122 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\" (UniqueName: \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\") pod \"a3ed995b-5678-4339-83a0-0c1e261995f7\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " Nov 01 21:57:21.572290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.572274 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\") pod \"a3ed995b-5678-4339-83a0-0c1e261995f7\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " Nov 01 21:57:21.572381 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.572302 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\") pod \"a3ed995b-5678-4339-83a0-0c1e261995f7\" (UID: \"a3ed995b-5678-4339-83a0-0c1e261995f7\") " Nov 01 21:57:21.572859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.572837 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca" pod "a3ed995b-5678-4339-83a0-0c1e261995f7" (UID: "a3ed995b-5678-4339-83a0-0c1e261995f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:21.575606 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.575582 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4" pod "a3ed995b-5678-4339-83a0-0c1e261995f7" (UID: "a3ed995b-5678-4339-83a0-0c1e261995f7"). InnerVolumeSpecName "kube-api-access-m4cb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:21.575981 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.575962 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca" pod "a3ed995b-5678-4339-83a0-0c1e261995f7" (UID: "a3ed995b-5678-4339-83a0-0c1e261995f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:21.673356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.673282 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:21.673356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.673315 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed995b-5678-4339-83a0-0c1e261995f7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:21.673356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:21.673325 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4cb4\" (UniqueName: \"kubernetes.io/projected/a3ed995b-5678-4339-83a0-0c1e261995f7-kube-api-access-m4cb4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:22.002769 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.002675 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fd51b98-183c-42be-a6d1-643ec4454d24" containerID="c822f41f19b1febb68c7e62630893c5e1eb9386e9df088c00b15d95f87604c8f" exitCode=0 Nov 01 21:57:22.003240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.002941 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" event={"ID":"2fd51b98-183c-42be-a6d1-643ec4454d24","Type":"ContainerDied","Data":"c822f41f19b1febb68c7e62630893c5e1eb9386e9df088c00b15d95f87604c8f"} Nov 01 21:57:22.034470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.033935 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" Nov 01 21:57:22.050844 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.050800 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g" event={"ID":"a3ed995b-5678-4339-83a0-0c1e261995f7","Type":"ContainerDied","Data":"7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8"} Nov 01 21:57:22.051022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.050907 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c86a0279c15c489a7eb46fb1ac3f8ad36da97f208cbde6d17ad7b252289dfb8" Nov 01 21:57:22.151488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.145709 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:22.151488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.145784 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:22.557471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.554077 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:22.557471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.554134 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:22.687488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:22.685141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:23.168469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:23.165438 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:23.168469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:23.165528 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:23.557477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:23.556955 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:23.557477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:23.557059 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:24.238484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.234411 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:24.238484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.234501 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:24.545474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.536399 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:24.572468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.567878 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:24.572468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.567941 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:24.616509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.601908 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\") pod \"2fd51b98-183c-42be-a6d1-643ec4454d24\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " Nov 01 21:57:24.616509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.602039 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\" (UniqueName: \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\") pod \"2fd51b98-183c-42be-a6d1-643ec4454d24\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " Nov 01 21:57:24.616509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.602093 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\") pod \"2fd51b98-183c-42be-a6d1-643ec4454d24\" (UID: \"2fd51b98-183c-42be-a6d1-643ec4454d24\") " Nov 01 21:57:24.647606 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.629141 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca" pod "2fd51b98-183c-42be-a6d1-643ec4454d24" (UID: "2fd51b98-183c-42be-a6d1-643ec4454d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:24.674152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.674102 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:24.711491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.676900 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m" pod "2fd51b98-183c-42be-a6d1-643ec4454d24" (UID: "2fd51b98-183c-42be-a6d1-643ec4454d24"). InnerVolumeSpecName "kube-api-access-7424m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:24.711491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.681623 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7424m\" (UniqueName: \"kubernetes.io/projected/2fd51b98-183c-42be-a6d1-643ec4454d24-kube-api-access-7424m\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:24.711491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.681657 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:24.734484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.723584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca" pod "2fd51b98-183c-42be-a6d1-643ec4454d24" (UID: "2fd51b98-183c-42be-a6d1-643ec4454d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:24.784406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.784360 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fd51b98-183c-42be-a6d1-643ec4454d24-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:24.798968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:24.798528 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.124465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.123419 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:25.124465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.123492 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:25.226380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.224425 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:25.402541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.350541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.552467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.537878 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" event={"ID":"2fd51b98-183c-42be-a6d1-643ec4454d24","Type":"ContainerDied","Data":"68a78f151911afe2e1498f653be94868cdfd6e8df333ed6c518070f550f129d0"} Nov 01 21:57:25.552467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.537962 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a78f151911afe2e1498f653be94868cdfd6e8df333ed6c518070f550f129d0" Nov 01 21:57:25.552467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.538057 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8" Nov 01 21:57:25.553717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.553686 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:25.553854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.553744 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:25.632433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.632388 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.738571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.738525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.859469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.821624 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.986599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.986489 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:57:26.064130 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:25.985234 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:26.136140 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.136098 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:26.151468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.136378 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:26.429567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.410569 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:26.429567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.423327 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:26.458212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.454416 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:26.579998 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.579961 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:26.580772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.580742 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:26.661597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.661550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:26.749138 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.749041 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:26.750020 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.749996 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:26.822547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.822507 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:26.823091 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:26.823072 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:27.013710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.001232 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:57:27.065478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.064209 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:27.065478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.064472 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:27.065478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.065085 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:27.069021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.068997 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:27.127542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.125539 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 21:57:27.127542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.125602 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 21:57:27.177685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.177658 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:27.188104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.188083 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:27.554248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.554209 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:57:27.554687 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.554271 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 21:57:27.838215 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:27.838116 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:28.550645 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.550589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:28.577198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.577167 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:28.580059 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.580036 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:28.593656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.593631 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 21:57:28.686599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.686569 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:28.687722 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.687700 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:28.689126 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:28.689078 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 21:57:29.052334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.052302 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp"] Nov 01 21:57:29.052960 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.052943 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a3ed995b-5678-4339-83a0-0c1e261995f7" containerName="revoke-keys" Nov 01 21:57:29.053012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.052962 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ed995b-5678-4339-83a0-0c1e261995f7" containerName="revoke-keys" Nov 01 21:57:29.053012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.052973 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fd51b98-183c-42be-a6d1-643ec4454d24" containerName="revoke-keys" Nov 01 21:57:29.053012 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.052978 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd51b98-183c-42be-a6d1-643ec4454d24" containerName="revoke-keys" Nov 01 21:57:29.053105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.053063 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ed995b-5678-4339-83a0-0c1e261995f7" containerName="revoke-keys" Nov 01 21:57:29.053105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.053071 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd51b98-183c-42be-a6d1-643ec4454d24" containerName="revoke-keys" Nov 01 21:57:29.056992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.056976 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.061479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.061435 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:57:29.061676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.061652 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:57:29.061676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.061624 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:57:29.061676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.061581 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:57:29.065837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.065813 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp"] Nov 01 21:57:29.150266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.150233 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4nz\" (UniqueName: \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.150423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.150408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.150494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.150473 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.251837 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.251809 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.252032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.251853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.252032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.251922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4nz\" (UniqueName: \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.252611 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.252584 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.254279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.254258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.264951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.264929 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4nz\" (UniqueName: \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\") pod \"maas-api-revoke-sub-0a2db6f5-8c5wp\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.373152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.373079 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:29.404771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.404693 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:29.693820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.693705 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp"] Nov 01 21:57:29.696633 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:29.696599 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9e484a_60b6_4a76_a125_0a5c7835b268.slice/crio-79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6 WatchSource:0}: Error finding container 79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6: Status 404 returned error can't find the container with id 79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6 Nov 01 21:57:29.806417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:29.806383 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:30.344905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:30.344865 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:30.612297 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:30.612216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" event={"ID":"6b9e484a-60b6-4a76-a125-0a5c7835b268","Type":"ContainerStarted","Data":"011b209a2d444ed822a96bd39891eba30e6b7562c3dd74a966956a40b9c870dc"} Nov 01 21:57:30.612297 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:30.612287 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" event={"ID":"6b9e484a-60b6-4a76-a125-0a5c7835b268","Type":"ContainerStarted","Data":"79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6"} Nov 01 21:57:30.637115 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:30.637070 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" podStartSLOduration=1.637056092 podStartE2EDuration="1.637056092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:30.632534284 +0000 UTC m=+804.325514773" watchObservedRunningTime="2026-11-01 21:57:30.637056092 +0000 UTC m=+804.330036928" Nov 01 21:57:31.258968 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:31.258930 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:31.623995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:31.623959 5036 generic.go:356] "Generic (PLEG): container finished" podID="6b9e484a-60b6-4a76-a125-0a5c7835b268" containerID="011b209a2d444ed822a96bd39891eba30e6b7562c3dd74a966956a40b9c870dc" exitCode=0 Nov 01 21:57:31.624191 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:31.624053 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" event={"ID":"6b9e484a-60b6-4a76-a125-0a5c7835b268","Type":"ContainerDied","Data":"011b209a2d444ed822a96bd39891eba30e6b7562c3dd74a966956a40b9c870dc"} Nov 01 21:57:32.239330 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.239290 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:32.804518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.804487 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:32.917835 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.917751 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\") pod \"6b9e484a-60b6-4a76-a125-0a5c7835b268\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " Nov 01 21:57:32.918003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.917854 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\" (UniqueName: \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\") pod \"6b9e484a-60b6-4a76-a125-0a5c7835b268\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " Nov 01 21:57:32.918003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.917934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\") pod \"6b9e484a-60b6-4a76-a125-0a5c7835b268\" (UID: \"6b9e484a-60b6-4a76-a125-0a5c7835b268\") " Nov 01 21:57:32.918182 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.918154 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca" pod "6b9e484a-60b6-4a76-a125-0a5c7835b268" (UID: "6b9e484a-60b6-4a76-a125-0a5c7835b268"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:32.918518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.918476 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:32.920067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.920045 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca" pod "6b9e484a-60b6-4a76-a125-0a5c7835b268" (UID: "6b9e484a-60b6-4a76-a125-0a5c7835b268"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:32.920195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.920176 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz" pod "6b9e484a-60b6-4a76-a125-0a5c7835b268" (UID: "6b9e484a-60b6-4a76-a125-0a5c7835b268"). InnerVolumeSpecName "kube-api-access-lr4nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:32.940877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:32.940847 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:33.020152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.020120 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4nz\" (UniqueName: \"kubernetes.io/projected/6b9e484a-60b6-4a76-a125-0a5c7835b268-kube-api-access-lr4nz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:33.020152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.020151 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9e484a-60b6-4a76-a125-0a5c7835b268-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:33.603491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.603438 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:33.642137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.642105 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" Nov 01 21:57:33.642137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.642127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp" event={"ID":"6b9e484a-60b6-4a76-a125-0a5c7835b268","Type":"ContainerDied","Data":"79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6"} Nov 01 21:57:33.642644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:33.642188 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79916a2782a16b444b7df4510fb786aaee42c1b3631091a625e7f3ab4c321ab6" Nov 01 21:57:34.522408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.522374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:34.529986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.529958 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 21:57:34.530652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.530634 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6b9e484a-60b6-4a76-a125-0a5c7835b268" containerName="revoke-keys" Nov 01 21:57:34.530759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.530654 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9e484a-60b6-4a76-a125-0a5c7835b268" containerName="revoke-keys" Nov 01 21:57:34.530857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.530844 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9e484a-60b6-4a76-a125-0a5c7835b268" containerName="revoke-keys" Nov 01 21:57:34.535663 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.535639 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.544556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.544039 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"distinct-2-w5-b0b7a3-kserve-self-signed-certs\"" Nov 01 21:57:34.547820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.547075 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 21:57:34.641142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.641086 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.641333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.641172 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.641333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.641204 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.641333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.641302 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.641581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.641381 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd96z\" (UniqueName: \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.642265 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.642235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.669849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.669810 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746256 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746313 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746423 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd96z\" (UniqueName: \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746597 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.746682 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.747163 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.747426 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.749540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.748758 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.773187 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.771986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd96z\" (UniqueName: \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.790512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.790459 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.791607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.791098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\") pod \"distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:34.854336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:34.854275 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:35.156501 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:35.156467 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode625cc85_0d4e_4677_a3ab_c05e6291d719.slice/crio-d3a3e5c6b19dcbb0e25235b5ac4e47fb03d8bd1cff7e151254a318b87c84a6e9 WatchSource:0}: Error finding container d3a3e5c6b19dcbb0e25235b5ac4e47fb03d8bd1cff7e151254a318b87c84a6e9: Status 404 returned error can't find the container with id d3a3e5c6b19dcbb0e25235b5ac4e47fb03d8bd1cff7e151254a318b87c84a6e9 Nov 01 21:57:35.157551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.157524 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 21:57:35.337672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.337633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:35.664130 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.664090 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" event={"ID":"e625cc85-0d4e-4677-a3ab-c05e6291d719","Type":"ContainerStarted","Data":"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424"} Nov 01 21:57:35.664643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.664146 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" event={"ID":"e625cc85-0d4e-4677-a3ab-c05e6291d719","Type":"ContainerStarted","Data":"d3a3e5c6b19dcbb0e25235b5ac4e47fb03d8bd1cff7e151254a318b87c84a6e9"} Nov 01 21:57:35.686231 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.686182 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podStartSLOduration=1.686169663 podStartE2EDuration="1.686169663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:35.683595406 +0000 UTC m=+809.376575895" watchObservedRunningTime="2026-11-01 21:57:35.686169663 +0000 UTC m=+809.379150176" Nov 01 21:57:35.854541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:35.854509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:36.122907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:36.122876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:36.838048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:36.838012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:37.622959 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.622919 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:37.871233 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.871196 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z"] Nov 01 21:57:37.876615 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.876549 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:37.882282 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.882257 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:57:37.883239 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.883223 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:57:37.884301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.884087 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:57:37.884301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.884185 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:57:37.893099 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:37.893070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z"] Nov 01 21:57:38.006487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.006437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrjg\" (UniqueName: \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.006652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.006520 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.006652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.006634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.108336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.108301 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrjg\" (UniqueName: \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.108521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.108392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.108584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.108533 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.109234 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.109206 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.110927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.110904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.121103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.121062 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrjg\" (UniqueName: \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\") pod \"maas-api-revoke-sub-c486a439-d6s2z\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.204244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.204175 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:38.449377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.449326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:38.498525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.498494 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z"] Nov 01 21:57:38.499020 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:38.498991 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe7304e_4df0_4429_9115_3e9958bbeb0f.slice/crio-1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286 WatchSource:0}: Error finding container 1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286: Status 404 returned error can't find the container with id 1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286 Nov 01 21:57:38.692786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:38.692750 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" event={"ID":"afe7304e-4df0-4429-9115-3e9958bbeb0f","Type":"ContainerStarted","Data":"1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286"} Nov 01 21:57:39.168134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.168044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:39.703082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.703048 5036 generic.go:356] "Generic (PLEG): container finished" podID="afe7304e-4df0-4429-9115-3e9958bbeb0f" containerID="b3fae844aa3c10604e7d493d72c92f0b5295e5c38b70792bc5d71aa27e12748a" exitCode=0 Nov 01 21:57:39.703262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.703112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" event={"ID":"afe7304e-4df0-4429-9115-3e9958bbeb0f","Type":"ContainerDied","Data":"b3fae844aa3c10604e7d493d72c92f0b5295e5c38b70792bc5d71aa27e12748a"} Nov 01 21:57:39.938556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.938466 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s"] Nov 01 21:57:39.942844 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.942821 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:39.952276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:39.952173 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s"] Nov 01 21:57:40.033379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.033348 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.033379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.033379 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.033622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.033549 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p68k\" (UniqueName: \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.123834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.123798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:40.147420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.147376 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.147660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.147430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.147660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.147564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p68k\" (UniqueName: \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.148340 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.148312 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.150085 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.150059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.161487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.161383 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p68k\" (UniqueName: \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\") pod \"maas-api-revoke-sub-96db6b56-df79s\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.263692 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.263602 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:40.561152 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:40.561120 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7faa87e3_4c15_4227_864b_614417a74ba7.slice/crio-0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c WatchSource:0}: Error finding container 0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c: Status 404 returned error can't find the container with id 0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c Nov 01 21:57:40.562095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.562069 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s"] Nov 01 21:57:40.714747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.714711 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" event={"ID":"7faa87e3-4c15-4227-864b-614417a74ba7","Type":"ContainerStarted","Data":"0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c"} Nov 01 21:57:40.863115 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.863089 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:40.910565 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.910532 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:40.960385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.960356 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\") pod \"afe7304e-4df0-4429-9115-3e9958bbeb0f\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " Nov 01 21:57:40.960575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.960499 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\" (UniqueName: \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\") pod \"afe7304e-4df0-4429-9115-3e9958bbeb0f\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " Nov 01 21:57:40.960575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.960568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\") pod \"afe7304e-4df0-4429-9115-3e9958bbeb0f\" (UID: \"afe7304e-4df0-4429-9115-3e9958bbeb0f\") " Nov 01 21:57:40.960947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.960910 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca" pod "afe7304e-4df0-4429-9115-3e9958bbeb0f" (UID: "afe7304e-4df0-4429-9115-3e9958bbeb0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:40.962639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.962606 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg" pod "afe7304e-4df0-4429-9115-3e9958bbeb0f" (UID: "afe7304e-4df0-4429-9115-3e9958bbeb0f"). InnerVolumeSpecName "kube-api-access-kwrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:40.962730 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:40.962644 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca" pod "afe7304e-4df0-4429-9115-3e9958bbeb0f" (UID: "afe7304e-4df0-4429-9115-3e9958bbeb0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:41.062104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.062032 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:41.062104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.062062 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrjg\" (UniqueName: \"kubernetes.io/projected/afe7304e-4df0-4429-9115-3e9958bbeb0f-kube-api-access-kwrjg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:41.062104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.062074 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe7304e-4df0-4429-9115-3e9958bbeb0f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:41.726918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.726884 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" Nov 01 21:57:41.727352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.726922 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z" event={"ID":"afe7304e-4df0-4429-9115-3e9958bbeb0f","Type":"ContainerDied","Data":"1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286"} Nov 01 21:57:41.727352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.726982 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd5e8fa78a97ddb35b0aa53ac2f71c447fd44f6ee670a3b3e579a772108f286" Nov 01 21:57:41.728950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.728886 5036 generic.go:356] "Generic (PLEG): container finished" podID="7faa87e3-4c15-4227-864b-614417a74ba7" containerID="28eef33569ba437c5a59ddd043312d5d442855ea6ddebd907af8c6771ce9044e" exitCode=0 Nov 01 21:57:41.728950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.728918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" event={"ID":"7faa87e3-4c15-4227-864b-614417a74ba7","Type":"ContainerDied","Data":"28eef33569ba437c5a59ddd043312d5d442855ea6ddebd907af8c6771ce9044e"} Nov 01 21:57:41.729191 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:41.729060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:42.422373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.422335 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:42.898952 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.898922 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:42.988370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.988292 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\") pod \"7faa87e3-4c15-4227-864b-614417a74ba7\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " Nov 01 21:57:42.988563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.988541 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\") pod \"7faa87e3-4c15-4227-864b-614417a74ba7\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " Nov 01 21:57:42.988622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.988571 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\" (UniqueName: \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\") pod \"7faa87e3-4c15-4227-864b-614417a74ba7\" (UID: \"7faa87e3-4c15-4227-864b-614417a74ba7\") " Nov 01 21:57:42.988980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.988955 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca" pod "7faa87e3-4c15-4227-864b-614417a74ba7" (UID: "7faa87e3-4c15-4227-864b-614417a74ba7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:42.989243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.989144 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:42.990870 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.990841 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca" pod "7faa87e3-4c15-4227-864b-614417a74ba7" (UID: "7faa87e3-4c15-4227-864b-614417a74ba7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:42.991052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:42.991031 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k" pod "7faa87e3-4c15-4227-864b-614417a74ba7" (UID: "7faa87e3-4c15-4227-864b-614417a74ba7"). InnerVolumeSpecName "kube-api-access-9p68k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:43.090308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.090276 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7faa87e3-4c15-4227-864b-614417a74ba7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:43.090308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.090305 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p68k\" (UniqueName: \"kubernetes.io/projected/7faa87e3-4c15-4227-864b-614417a74ba7-kube-api-access-9p68k\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:43.133864 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.133829 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:43.501185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.501143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:43.630977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.630936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:43.745356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.745323 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" Nov 01 21:57:43.745356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.745349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s" event={"ID":"7faa87e3-4c15-4227-864b-614417a74ba7","Type":"ContainerDied","Data":"0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c"} Nov 01 21:57:43.745627 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.745425 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fcaae1bf68b10769ebcac53f8e372b313bac3cb986f00fc1021edc3b250280c" Nov 01 21:57:43.918386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:43.918347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:44.728296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:44.728264 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:44.855002 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:44.854975 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:44.868259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:44.868234 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:45.513565 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.513521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:45.605119 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.605091 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt"] Nov 01 21:57:45.606277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606252 5036 cpu_manager.go:414] "containerMap: removing container" podUID="afe7304e-4df0-4429-9115-3e9958bbeb0f" containerName="revoke-keys" Nov 01 21:57:45.606277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606273 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe7304e-4df0-4429-9115-3e9958bbeb0f" containerName="revoke-keys" Nov 01 21:57:45.606487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606298 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7faa87e3-4c15-4227-864b-614417a74ba7" containerName="revoke-keys" Nov 01 21:57:45.606487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606306 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faa87e3-4c15-4227-864b-614417a74ba7" containerName="revoke-keys" Nov 01 21:57:45.606487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606480 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe7304e-4df0-4429-9115-3e9958bbeb0f" containerName="revoke-keys" Nov 01 21:57:45.606644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.606496 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7faa87e3-4c15-4227-864b-614417a74ba7" containerName="revoke-keys" Nov 01 21:57:45.610672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.610653 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.615773 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.615099 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:57:45.615963 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.615737 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:57:45.618897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.618849 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:57:45.618989 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.618913 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:57:45.622806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.622783 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt"] Nov 01 21:57:45.725687 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.725656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw655\" (UniqueName: \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.725859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.725708 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.725859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.725737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.774744 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.774720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:45.827611 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.827576 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw655\" (UniqueName: \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.827770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.827656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.827770 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.827690 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.829225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.829201 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.831643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.831592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.840276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.840257 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw655\" (UniqueName: \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\") pod \"maas-api-revoke-sub-d1908b23-z44dt\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:45.869475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.868227 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn"] Nov 01 21:57:45.875987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.875963 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:45.893061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.891874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn"] Nov 01 21:57:45.929357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:45.929326 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:46.030888 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.030819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmcw\" (UniqueName: \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.031037 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.030982 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.031095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.031053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.132810 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.132769 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.132810 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.132811 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.132996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.132988 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmcw\" (UniqueName: \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.133597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.133572 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.135161 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.135139 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.145420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.145395 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmcw\" (UniqueName: \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\") pod \"maas-api-revoke-sub-63dbe56e-qprbn\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.218665 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.218635 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:46.225096 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:46.225070 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda2d64a_ca89_433b_8ff5_89ace6b52474.slice/crio-7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7 WatchSource:0}: Error finding container 7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7: Status 404 returned error can't find the container with id 7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7 Nov 01 21:57:46.225974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.225946 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt"] Nov 01 21:57:46.364733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.364690 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:46.510652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.510625 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn"] Nov 01 21:57:46.772813 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.772779 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" event={"ID":"be56b6f0-adae-4bea-acef-da745eadd701","Type":"ContainerStarted","Data":"daae0dc5c3bf963066cbedd3f4349f74718a121b09cdcc9d931109d0528cf6e4"} Nov 01 21:57:46.774496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.774467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" event={"ID":"bda2d64a-ca89-433b-8ff5-89ace6b52474","Type":"ContainerStarted","Data":"4d2c654a39dd9329cdeed2decd073644187440acf87b3444196d5f179d4b26dc"} Nov 01 21:57:46.774637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.774519 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" event={"ID":"bda2d64a-ca89-433b-8ff5-89ace6b52474","Type":"ContainerStarted","Data":"7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7"} Nov 01 21:57:46.794700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:46.794613 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" podStartSLOduration=1.794601762 podStartE2EDuration="1.794601762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:46.792641648 +0000 UTC m=+820.485622134" watchObservedRunningTime="2026-11-01 21:57:46.794601762 +0000 UTC m=+820.487582248" Nov 01 21:57:47.068956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.068862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:47.741601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.741568 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:47.786882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.786838 5036 generic.go:356] "Generic (PLEG): container finished" podID="be56b6f0-adae-4bea-acef-da745eadd701" containerID="5a729ba27b1143acc242b320252ad5d7bb55b3d3a7cb77643b30cc1ab21eaa2a" exitCode=0 Nov 01 21:57:47.787344 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.786986 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" event={"ID":"be56b6f0-adae-4bea-acef-da745eadd701","Type":"ContainerDied","Data":"5a729ba27b1143acc242b320252ad5d7bb55b3d3a7cb77643b30cc1ab21eaa2a"} Nov 01 21:57:47.789681 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.789640 5036 generic.go:356] "Generic (PLEG): container finished" podID="bda2d64a-ca89-433b-8ff5-89ace6b52474" containerID="4d2c654a39dd9329cdeed2decd073644187440acf87b3444196d5f179d4b26dc" exitCode=0 Nov 01 21:57:47.789846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.789694 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" event={"ID":"bda2d64a-ca89-433b-8ff5-89ace6b52474","Type":"ContainerDied","Data":"4d2c654a39dd9329cdeed2decd073644187440acf87b3444196d5f179d4b26dc"} Nov 01 21:57:47.894182 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.894102 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 21:57:47.899577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.899471 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:47.919373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.919347 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 21:57:47.946410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.946380 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 21:57:47.952364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.951619 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:47.972586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.972560 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 21:57:47.986166 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.986139 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 21:57:47.991710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.991686 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:47.997107 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:47.997080 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f033b1-openshift-default-dockercfg-svg6v\"" Nov 01 21:57:48.006566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.004968 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 21:57:48.057486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.057621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5gw\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.057621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057545 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.057621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057563 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.057739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.057739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057725 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.057865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057838 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.057913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057896 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.057951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.057995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.057981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058045 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.058079 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058064 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.058110 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058098 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058199 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jsv\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058307 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.058339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058315 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058333 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058402 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058433 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058414 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058478 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.058517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058536 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h94c\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.058584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058555 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058586 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.058652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058603 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.058652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.058625 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.096069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.095730 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 21:57:48.110913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.107167 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.122399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.122369 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 21:57:48.159925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.159925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jsv\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159935 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159951 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.159995 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160082 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160102 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h94c\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160116 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160134 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160157 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160241 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160261 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5gw\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160276 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160324 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160343 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160366 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160586 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160612 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.160680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160665 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.161165 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.161165 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.160886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.161938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.161363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.161938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.161568 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.161938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.161688 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.162173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.161965 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.162173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.162102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.163674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.162859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.163674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.163131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.163674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.163356 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.163674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.163592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.163674 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.163637 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.164286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.164261 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.165634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.165614 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.165733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.165642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.165733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.165657 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.166168 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.166150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.166694 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.166670 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.166929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.166911 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.167573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.167550 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.175983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.175105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.175983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.175593 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h94c\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.175983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.175622 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jsv\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.175983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.175933 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.176851 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.176827 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.176982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.176967 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5gw\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.215523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.215493 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:48.262162 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262129 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262173 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262268 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc7z\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262294 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262415 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262756 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262732 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.262849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.262837 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.276289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.276113 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:48.288804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.288779 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 21:57:48.317352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.317312 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 21:57:48.317541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.317480 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.331889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.331680 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:48.367268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367344 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367381 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367420 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc7z\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367618 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367734 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367735 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.367790 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.367767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.368666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.368602 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.370337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.370310 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.370641 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.370616 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.370899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.370876 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.371151 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.371128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.372938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.372917 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.384049 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.383997 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.386921 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.386783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc7z\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.388228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.388159 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.435949 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.435136 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.468756 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.468818 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.468852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5qq\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.468949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.468980 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.469020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.469045 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.469154 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.469399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.469202 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.509144 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.509108 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 21:57:48.509843 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:48.509816 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a17808_2855_4640_9c6f_a41121f028f0.slice/crio-cf64d1dd211bfe94e7e6845968db7323cc21c4f120df3fdc194a4feda08a1df7 WatchSource:0}: Error finding container cf64d1dd211bfe94e7e6845968db7323cc21c4f120df3fdc194a4feda08a1df7: Status 404 returned error can't find the container with id cf64d1dd211bfe94e7e6845968db7323cc21c4f120df3fdc194a4feda08a1df7 Nov 01 21:57:48.512067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.511930 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.512148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.512101 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.512193 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.512150 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.570857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.570776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571692 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571736 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5qq\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571879 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571926 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.571992 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.572152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.572229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.573709 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.573957 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.574028 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.575047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.574531 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.576013 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.575970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.577856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.577104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.580072 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.579938 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.586246 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.586203 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.587551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.587517 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5qq\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.602276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.602253 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 21:57:48.605006 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:48.604981 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c64037_3253_45bf_8499_8dda6bdb0437.slice/crio-b9946d2d493f1eafe378c633476836dbe7f059983dcbe477d82d7b102fdd656c WatchSource:0}: Error finding container b9946d2d493f1eafe378c633476836dbe7f059983dcbe477d82d7b102fdd656c: Status 404 returned error can't find the container with id b9946d2d493f1eafe378c633476836dbe7f059983dcbe477d82d7b102fdd656c Nov 01 21:57:48.608318 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.608283 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.608408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.608348 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.608408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.608378 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.657945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.657917 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:48.661140 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.661110 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 21:57:48.665506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.665148 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.665506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.665207 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.665506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.665235 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.766476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.756788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:48.809477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.809428 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 21:57:48.820380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.820345 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" event={"ID":"b4c5dadb-6e55-4c39-927a-3742c70b1584","Type":"ContainerStarted","Data":"bc3c426b90044e818f2fd2de49740d52fe4137ed5f15405d649dcf0631fe81a7"} Nov 01 21:57:48.828395 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.828365 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" event={"ID":"e3c64037-3253-45bf-8499-8dda6bdb0437","Type":"ContainerStarted","Data":"297d33e9081a19351fdc600901a0f74cb284b12382b875771ef27dae8050d094"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.846028 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" event={"ID":"e3c64037-3253-45bf-8499-8dda6bdb0437","Type":"ContainerStarted","Data":"b9946d2d493f1eafe378c633476836dbe7f059983dcbe477d82d7b102fdd656c"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.846751 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" event={"ID":"18a17808-2855-4640-9c6f-a41121f028f0","Type":"ContainerStarted","Data":"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.846805 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" event={"ID":"18a17808-2855-4640-9c6f-a41121f028f0","Type":"ContainerStarted","Data":"cf64d1dd211bfe94e7e6845968db7323cc21c4f120df3fdc194a4feda08a1df7"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.849802 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.849898 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:48.898432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:48.849944 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:49.071126 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.071048 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podStartSLOduration=2.071028442 podStartE2EDuration="2.071028442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:49.066487219 +0000 UTC m=+822.759467706" watchObservedRunningTime="2026-11-01 21:57:49.071028442 +0000 UTC m=+822.764008929" Nov 01 21:57:49.135926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.135848 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podStartSLOduration=2.13582596 podStartE2EDuration="2.13582596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:49.093859701 +0000 UTC m=+822.786840200" watchObservedRunningTime="2026-11-01 21:57:49.13582596 +0000 UTC m=+822.828806448" Nov 01 21:57:49.254819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.253564 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:57:49.286185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.280814 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:57:49.306682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.306635 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:49.306958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.306912 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:49.339073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.328088 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:49.339073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.328137 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:49.779474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.697136 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 21:57:49.779474 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:57:49.730659 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b12813_1b64_4e6a_844f_ca3a13af79ec.slice/crio-c2ecae6b012607c8e1de390357e55ea274568547c57fbaef48f158b812485ebd WatchSource:0}: Error finding container c2ecae6b012607c8e1de390357e55ea274568547c57fbaef48f158b812485ebd: Status 404 returned error can't find the container with id c2ecae6b012607c8e1de390357e55ea274568547c57fbaef48f158b812485ebd Nov 01 21:57:49.779474 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:57:49.776964 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:57:49.829918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.780651 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:49.831123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.831072 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:49.855803 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:49.855751 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:57:50.366733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.344199 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:50.366733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.344266 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:50.366733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.346726 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:50.366733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.346784 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:50.669666 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.669554 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:50.769716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.769682 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:50.777650 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.777617 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:50.974831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.974744 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\") pod \"bda2d64a-ca89-433b-8ff5-89ace6b52474\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " Nov 01 21:57:51.113658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.113622 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\" (UniqueName: \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\") pod \"be56b6f0-adae-4bea-acef-da745eadd701\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " Nov 01 21:57:51.113909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.113886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\") pod \"be56b6f0-adae-4bea-acef-da745eadd701\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " Nov 01 21:57:51.139651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.139619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\") pod \"be56b6f0-adae-4bea-acef-da745eadd701\" (UID: \"be56b6f0-adae-4bea-acef-da745eadd701\") " Nov 01 21:57:51.139940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.139919 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\" (UniqueName: \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\") pod \"bda2d64a-ca89-433b-8ff5-89ace6b52474\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " Nov 01 21:57:51.140056 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.140042 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\") pod \"bda2d64a-ca89-433b-8ff5-89ace6b52474\" (UID: \"bda2d64a-ca89-433b-8ff5-89ace6b52474\") " Nov 01 21:57:51.152470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.024618 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca" pod "bda2d64a-ca89-433b-8ff5-89ace6b52474" (UID: "bda2d64a-ca89-433b-8ff5-89ace6b52474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:51.152711 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.146895 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca" pod "be56b6f0-adae-4bea-acef-da745eadd701" (UID: "be56b6f0-adae-4bea-acef-da745eadd701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:57:51.152834 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:50.999320 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" event={"ID":"11b12813-1b64-4e6a-844f-ca3a13af79ec","Type":"ContainerStarted","Data":"c2ecae6b012607c8e1de390357e55ea274568547c57fbaef48f158b812485ebd"} Nov 01 21:57:51.152997 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.152973 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" event={"ID":"b4c5dadb-6e55-4c39-927a-3742c70b1584","Type":"ContainerStarted","Data":"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8"} Nov 01 21:57:51.170339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.170307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" event={"ID":"b37a2f2d-e9f4-45c3-b081-e0213124dfac","Type":"ContainerStarted","Data":"9f456919df5aace9f67af5f3084e4e5fc9e20ebc59bda16d5320b3c75c6f4ae9"} Nov 01 21:57:51.189537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.187701 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.308669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.241643 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca" pod "bda2d64a-ca89-433b-8ff5-89ace6b52474" (UID: "bda2d64a-ca89-433b-8ff5-89ace6b52474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:51.308669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.242202 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:51.308669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.242257 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:51.308669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.276497 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:51.308669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.276551 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:51.338931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.338512 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca" pod "be56b6f0-adae-4bea-acef-da745eadd701" (UID: "be56b6f0-adae-4bea-acef-da745eadd701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:57:51.339088 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.339067 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:51.339150 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.339119 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:51.339223 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.339188 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:57:51.339943 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.339890 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw" pod "be56b6f0-adae-4bea-acef-da745eadd701" (UID: "be56b6f0-adae-4bea-acef-da745eadd701"). InnerVolumeSpecName "kube-api-access-8gmcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:51.351987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.349891 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.351987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.349977 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:57:51.370467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.358099 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655" pod "bda2d64a-ca89-433b-8ff5-89ace6b52474" (UID: "bda2d64a-ca89-433b-8ff5-89ace6b52474"). InnerVolumeSpecName "kube-api-access-kw655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386285 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw655\" (UniqueName: \"kubernetes.io/projected/bda2d64a-ca89-433b-8ff5-89ace6b52474-kube-api-access-kw655\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386319 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386335 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda2d64a-ca89-433b-8ff5-89ace6b52474-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386351 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmcw\" (UniqueName: \"kubernetes.io/projected/be56b6f0-adae-4bea-acef-da745eadd701-kube-api-access-8gmcw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386367 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.388480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.386381 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be56b6f0-adae-4bea-acef-da745eadd701-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:57:51.402944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.402895 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.441482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.440600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:57:51.454933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.454584 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:51.454933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.454651 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:51.483471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.455710 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podStartSLOduration=4.455690978 podStartE2EDuration="4.455690978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:51.371663597 +0000 UTC m=+825.064644086" watchObservedRunningTime="2026-11-01 21:57:51.455690978 +0000 UTC m=+825.148671465" Nov 01 21:57:51.483471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.469129 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.538471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.516091 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podStartSLOduration=3.516072755 podStartE2EDuration="3.516072755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:51.428178451 +0000 UTC m=+825.121158941" watchObservedRunningTime="2026-11-01 21:57:51.516072755 +0000 UTC m=+825.209053242" Nov 01 21:57:51.664828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.664722 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.696480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.692745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:51.696480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.693424 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:57:51.797672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.783699 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:51.941359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.940643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.002494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:51.986638 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.400876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.282629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:52.400876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.283378 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:57:52.400876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.348045 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:52.400876 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.348097 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:52.431686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.402533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.556847 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.431986 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:57:52.597510 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.597434 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:57:52.599116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.447969 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:52.638873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.638830 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.610540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.476526 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.448019 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.641096 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.480125 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:52.664392 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.641150 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:52.698517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.698474 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.014046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.784259 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:57:53.014897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.014863 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" event={"ID":"11b12813-1b64-4e6a-844f-ca3a13af79ec","Type":"ContainerStarted","Data":"28785ef7ebdc8dfc27bb9f90e569fb644e0b4327743dca70e9885db2fddcaad6"} Nov 01 21:57:53.015107 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.015092 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:57:53.015232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.015222 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:57:53.147062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.008661 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.147062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:52.961564 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.147062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.041237 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:57:53.147062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.041294 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:57:53.147062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.140856 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.283528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.279169 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.345138 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:53.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.379709 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.384238 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:53.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.287163 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.561648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.287029 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.915705 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.915640 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:53.916019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.569097 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 01 21:57:53.916179 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.569122 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:57:53.916676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.568963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:57:53.917143 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.805068 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.917335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.917287 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:53.922651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.837705 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:53.923327 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.860955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.941472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.860913 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.945468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.945092 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:53.945468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.685906 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" event={"ID":"be56b6f0-adae-4bea-acef-da745eadd701","Type":"ContainerDied","Data":"daae0dc5c3bf963066cbedd3f4349f74718a121b09cdcc9d931109d0528cf6e4"} Nov 01 21:57:53.945468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.945260 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daae0dc5c3bf963066cbedd3f4349f74718a121b09cdcc9d931109d0528cf6e4" Nov 01 21:57:53.945468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.945290 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:57:53.945468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.884435 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:57:53.945859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.651764 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:57:53.946461 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.687120 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn" Nov 01 21:57:53.960886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.960857 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:57:53.961246 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.961214 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:57:54.065242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:53.995501 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:54.065242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.060974 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.065242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.061408 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.089476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.085910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" event={"ID":"b37a2f2d-e9f4-45c3-b081-e0213124dfac","Type":"ContainerStarted","Data":"e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318"} Nov 01 21:57:54.157654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.157548 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.233617 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.233566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:57:54.234364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.234337 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:57:54.469713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.469620 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.469989 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.469957 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:54.470194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.470176 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.470320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.470302 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:54.497965 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.497921 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:54.607956 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.607913 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 21:57:54.699470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.677248 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.699470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.695961 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:54.852521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.852477 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.852883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.852845 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:54.926811 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.864948 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.927897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.927370 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.928237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.928167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:54.945634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.945588 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.946016 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:54.945988 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.103684 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.103576 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.192469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.138070 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:57:55.288471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.236734 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" Nov 01 21:57:55.288471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.270770 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.288471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.270886 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:55.288471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.285044 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.321903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.321857 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:57:55.455521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.330670 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:57:55.503464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.330745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:55.571509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.550653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.571728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.370048 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.571800 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.571712 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:55.575488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.546859 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.575488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.572000 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:55.575488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.546933 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.575488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.572071 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:55.590468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.578582 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.590468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.578648 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:55.671466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.385717 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:55.700954 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:57:55.639408 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s" Nov 01 21:57:55.701264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.701227 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt" event={"ID":"bda2d64a-ca89-433b-8ff5-89ace6b52474","Type":"ContainerDied","Data":"7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7"} Nov 01 21:57:55.701417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.701401 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df8ec9b125502c48f8d45ca807eb39c81703fa451aa5366dfcdcf1de40742e7" Nov 01 21:57:55.701632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.701616 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:57:55.819725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.819691 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.820481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.820438 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:55.837098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.837020 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podStartSLOduration=7.836982638 podStartE2EDuration="7.836982638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:53.332732805 +0000 UTC m=+827.025713287" watchObservedRunningTime="2026-11-01 21:57:55.836982638 +0000 UTC m=+829.529963128" Nov 01 21:57:55.954611 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.954562 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.969600 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.969550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.995468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.970093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:55.996808 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:55.978211 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:57:56.074540 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.049527 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.151469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.150602 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.270806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.241244 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:57:56.297740 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.271133 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.297740 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.271192 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:56.297740 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.275523 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.298830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.298137 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.325218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.324566 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.325218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.324628 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:56.391584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.367719 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.391584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.367784 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:56.425601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.407039 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.590389 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.566499 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:56.635521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.579522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:57:56.635521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.579562 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:56.740469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.712395 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.740469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.712472 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:56.741600 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.741568 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.741803 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.741777 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:56.863157 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.863061 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.925182 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79"] Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926112 5036 cpu_manager.go:414] "containerMap: removing container" podUID="be56b6f0-adae-4bea-acef-da745eadd701" containerName="revoke-keys" Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926133 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be56b6f0-adae-4bea-acef-da745eadd701" containerName="revoke-keys" Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926173 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bda2d64a-ca89-433b-8ff5-89ace6b52474" containerName="revoke-keys" Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926182 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda2d64a-ca89-433b-8ff5-89ace6b52474" containerName="revoke-keys" Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926370 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="be56b6f0-adae-4bea-acef-da745eadd701" containerName="revoke-keys" Nov 01 21:57:56.936467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.926391 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda2d64a-ca89-433b-8ff5-89ace6b52474" containerName="revoke-keys" Nov 01 21:57:56.965479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.963265 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:57:56.973619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.968434 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:56.973619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.970414 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:57.010199 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.986840 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:57:57.010199 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.986959 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:57:57.010199 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:56.987018 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:57:57.010199 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.003366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.010199 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.003990 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:57.030486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.015180 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.030486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.015349 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.030486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.015514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sft8g\" (UniqueName: \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.075477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.075058 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:57:57.098467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.077619 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79"] Nov 01 21:57:57.123575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.123017 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.123575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.123176 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.123575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.123300 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sft8g\" (UniqueName: \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.125605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.125579 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.157196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.157157 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:57:57.165857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.165824 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sft8g\" (UniqueName: \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.218474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.211880 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e56254-tsq79\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.218474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.217363 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.218474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.217410 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:57.247788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.246974 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.288269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.287558 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.288269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.287616 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:57.312477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.310563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.338070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.338037 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.338427 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.338352 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:57.408227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.408075 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.435885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.435843 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.436171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.436138 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:57.502786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.502743 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:57:57.592734 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.592692 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:57.710527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.682562 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.710527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.682718 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:57.734474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.712569 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.734474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.712926 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.928466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.925745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:58.002509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.974843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:58.002509 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:57.976796 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.023496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.014751 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.023496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.015121 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.069487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.027779 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9"] Nov 01 21:57:58.069487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.057878 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 21:57:58.099488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.083940 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.217505 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.217375 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.246520 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.243587 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.246520 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.243646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:58.251264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.250511 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:57:58.278273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.278162 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.278273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.278218 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:58.295850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.295766 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9"] Nov 01 21:57:58.318984 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.314523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.372473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.347156 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqd8\" (UniqueName: \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.372473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.347226 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.372473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.347277 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.372473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.347720 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.372473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.347768 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:58.410610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.410528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.463843 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.458566 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.463843 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.458639 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:58.465409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.465352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqd8\" (UniqueName: \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.466081 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.466039 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.483134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.483097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.484429 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.484304 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.500729 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.500581 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 01 21:57:58.584933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.582213 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:58.606875 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.596212 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.691502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.665182 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.691502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.665252 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:58.745475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.720822 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.749471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.725639 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:57:59.039545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:58.980744 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.039545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.037830 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:59.123501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.099587 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqd8\" (UniqueName: \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\") pod \"maas-api-revoke-sub-91ee82a6-q8sg9\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:59.123501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.099802 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.163470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.124539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:57:59.200788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.200754 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.225526 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.288589 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.338343 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.305676 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79"] Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.317598 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.225652 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.339462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.261044 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.372464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.343664 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:57:59.428672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.426546 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.428672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.426602 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:57:59.428823 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.428675 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:57:59.445613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.445580 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.449569 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.445837 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:57:59.492535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.468651 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" event={"ID":"787a27a9-8bce-4235-b2f3-f865827ecd90","Type":"ContainerStarted","Data":"54efa380580e9302e35af185813aa4001a5d8e3ccef2790afeb2b585387031cb"} Nov 01 21:57:59.598193 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.598106 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:57:59.674471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.670587 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.679466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.675209 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:57:59.747984 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.747941 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.755175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:57:59.747889 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.023682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.019478 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.047004 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.046964 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:00.081475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.074994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.200007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.229816 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.231874 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.231920 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.301381 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.301464 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.355536 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.355626 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.356513 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:00.371555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.356831 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:00.504130 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.447749 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.656524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.654911 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.709708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.656754 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.709708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.656802 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:00.709708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.664752 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.709708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.684582 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.709708 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.684642 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:00.910077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.821577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.910077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.822242 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:00.910077 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:00.858868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:58:01.022639 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.022529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.077922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.077878 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:01.155470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.146309 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n"] Nov 01 21:58:01.205376 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.203115 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:01.224468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.221786 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.224468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.221839 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:58:01.232974 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.229933 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:01.233235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.233169 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.312469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.311582 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.312469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.311660 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:58:01.352824 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.352792 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n"] Nov 01 21:58:01.353043 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.353022 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" event={"ID":"660aaf4f-4ed4-4780-8571-9727560319bb","Type":"ContainerStarted","Data":"d78f3620ceb4e4ee736bdbaf4ee7120ff4638d199720a89ad77ba673c4189235"} Nov 01 21:58:01.353153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.353138 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" event={"ID":"787a27a9-8bce-4235-b2f3-f865827ecd90","Type":"ContainerStarted","Data":"57f2333358f40c92c67d7fb9e4fd2e483e2059be8b0fdb6bef64e89e571644f4"} Nov 01 21:58:01.367526 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.353332 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.373575 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.361148 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.377539 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.377295 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" podStartSLOduration=6.377278023 podStartE2EDuration="6.377278023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:01.32102771 +0000 UTC m=+835.014008200" watchObservedRunningTime="2026-11-01 21:58:01.377278023 +0000 UTC m=+835.070258511" Nov 01 21:58:01.377730 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.374098 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:01.381471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.361211 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.393542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.389110 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9"] Nov 01 21:58:01.393542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.361481 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:01.437478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.427560 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.437478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.427612 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.437478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.427715 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.454477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.450520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:01.457522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.455110 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.457522 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.455155 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:01.541786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.532618 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.541786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.532670 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.541786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.532763 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.541786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.536294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.591594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.560866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.661817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.660680 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.691074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.690979 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.693626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.692939 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\") pod \"maas-api-revoke-sub-f49fdb93-xkq2n\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.693626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.693408 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.693626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.693475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:01.768262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.766958 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:01.860597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.860550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.861874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.861530 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:01.861874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:01.861657 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.032635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.032591 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:02.110466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.109555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.203757 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.203706 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.217829 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.217793 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.218096 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.218066 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:58:02.246573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.246527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:02.279881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.279848 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.280152 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.280110 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:58:02.357472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.355394 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.357472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.355464 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:02.455497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.403704 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:02.455497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.404092 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:02.455497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.436839 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.455497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.436890 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:02.522527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.497554 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.695831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.663971 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.695831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.664030 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:02.695831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.664154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:02.695831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.664272 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.787978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.773086 5036 generic.go:356] "Generic (PLEG): container finished" podID="787a27a9-8bce-4235-b2f3-f865827ecd90" containerID="57f2333358f40c92c67d7fb9e4fd2e483e2059be8b0fdb6bef64e89e571644f4" exitCode=0 Nov 01 21:58:02.787978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.773412 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" event={"ID":"787a27a9-8bce-4235-b2f3-f865827ecd90","Type":"ContainerDied","Data":"57f2333358f40c92c67d7fb9e4fd2e483e2059be8b0fdb6bef64e89e571644f4"} Nov 01 21:58:02.927966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.926643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:58:02.927966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.927029 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:02.927966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:02.927697 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.045812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.045767 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.180470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.143919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:03.223986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.187293 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n"] Nov 01 21:58:03.223986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.217594 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:58:03.223986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.218855 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.281762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.241587 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.281762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.241638 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:58:03.281762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.249520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:03.304945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.302882 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:58:03.304945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.303909 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.304945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.303971 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:58:03.339061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.333690 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:58:03.339061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.335681 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.339061 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.335723 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:03.435486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.431530 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.435486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.432335 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:03.451478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.436529 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:58:03.457221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.456737 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.457221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.456786 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:03.503637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.503583 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:03.608062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.605562 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.673574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.669954 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.673574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.670029 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:03.673574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.671809 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.712481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:03.711676 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:58:04.070660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.067797 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:58:04.070660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.068614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:04.070660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.069395 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:04.144555 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.144438 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.199470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.169006 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:04.347500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.213801 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.347500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.233735 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.347500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.347486 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 01 21:58:04.347814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.295213 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.347814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.235490 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.348022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.284816 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.348114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.348048 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 21:58:04.348114 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.316912 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.425950 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.425854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" event={"ID":"660aaf4f-4ed4-4780-8571-9727560319bb","Type":"ContainerStarted","Data":"4d7d6a6abe18c375e0173e4a87356cbadb73ac6131b18a089acba2bd9906a6fe"} Nov 01 21:58:04.523333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.504018 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.523333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.518747 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.523333 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.518815 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:04.543420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.542195 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.543420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.542263 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:04.544748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.544718 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.545980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.545310 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:04.566366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.566330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:04.682805 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.677361 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.723541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.714039 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.723541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:04.714135 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:05.065250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.064437 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:05.065250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.064558 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" event={"ID":"f467320f-9bc7-47a4-962c-1769d84a1aae","Type":"ContainerStarted","Data":"f4bd541651333602317911161deb35620571275728fd6f528be272ac34ccd306"} Nov 01 21:58:05.065250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.064632 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:58:05.065250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.064694 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:58:05.069953 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.069919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:05.272678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.272625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:05.274250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.274220 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Nov 01 21:58:05.354652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.353858 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:05.361334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.358652 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:05.361334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.360162 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:05.404467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.369903 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:05.404765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.404724 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:05.520173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.509736 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:05.520173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.509798 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:05.588010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.587907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:05.613068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.588582 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:05.614170 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.588629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:05.617171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.617144 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:05.646472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.644420 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:58:05.680320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.680276 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 21:58:05.718483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.709685 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 01 21:58:06.062125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.062092 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 01 21:58:06.062279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.963001 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:58:06.155286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.155258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:58:06.158551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.158528 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:58:06.177909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.177576 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:06.232326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.232251 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:58:06.232326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.232322 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 21:58:06.235726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.235704 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:58:06.262812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.262789 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:58:06.270336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:05.818585 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:06.279728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.072539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:06.295689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.295642 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:06.300911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.300624 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:58:06.335843 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.306379 5036 generic.go:356] "Generic (PLEG): container finished" podID="660aaf4f-4ed4-4780-8571-9727560319bb" containerID="4d7d6a6abe18c375e0173e4a87356cbadb73ac6131b18a089acba2bd9906a6fe" exitCode=0 Nov 01 21:58:06.335843 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.306596 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" event={"ID":"660aaf4f-4ed4-4780-8571-9727560319bb","Type":"ContainerDied","Data":"4d7d6a6abe18c375e0173e4a87356cbadb73ac6131b18a089acba2bd9906a6fe"} Nov 01 21:58:06.338415 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.338391 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:58:06.352394 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.352369 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 21:58:06.352730 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.352659 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 21:58:06.376105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.376077 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:58:06.376753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.376729 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:58:06.400785 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.398255 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\" (UniqueName: \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\") pod \"787a27a9-8bce-4235-b2f3-f865827ecd90\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " Nov 01 21:58:06.401040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.401002 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\") pod \"787a27a9-8bce-4235-b2f3-f865827ecd90\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " Nov 01 21:58:06.401356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.401337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\") pod \"787a27a9-8bce-4235-b2f3-f865827ecd90\" (UID: \"787a27a9-8bce-4235-b2f3-f865827ecd90\") " Nov 01 21:58:06.402880 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.402853 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca" pod "787a27a9-8bce-4235-b2f3-f865827ecd90" (UID: "787a27a9-8bce-4235-b2f3-f865827ecd90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:06.441020 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.440077 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca" pod "787a27a9-8bce-4235-b2f3-f865827ecd90" (UID: "787a27a9-8bce-4235-b2f3-f865827ecd90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:06.442501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.442471 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 01 21:58:06.442627 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.442527 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 01 21:58:06.444370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.443914 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g" pod "787a27a9-8bce-4235-b2f3-f865827ecd90" (UID: "787a27a9-8bce-4235-b2f3-f865827ecd90"). InnerVolumeSpecName "kube-api-access-sft8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:06.477500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.477472 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:06.513964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.513072 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:06.514690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.514666 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sft8g\" (UniqueName: \"kubernetes.io/projected/787a27a9-8bce-4235-b2f3-f865827ecd90-kube-api-access-sft8g\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:06.514871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:06.514855 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/787a27a9-8bce-4235-b2f3-f865827ecd90-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:07.193139 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:07.193098 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf467320f_9bc7_47a4_962c_1769d84a1aae.slice/crio-conmon-dacce5b33d66817ecf0a49f86de0efa9a8a4b291b7642c156ab03c8550a9cb55.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf467320f_9bc7_47a4_962c_1769d84a1aae.slice/crio-dacce5b33d66817ecf0a49f86de0efa9a8a4b291b7642c156ab03c8550a9cb55.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:58:07.256168 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.256131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:07.385846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.385811 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:58:07.393499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.386645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" event={"ID":"660aaf4f-4ed4-4780-8571-9727560319bb","Type":"ContainerDied","Data":"d78f3620ceb4e4ee736bdbaf4ee7120ff4638d199720a89ad77ba673c4189235"} Nov 01 21:58:07.393499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.386740 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d78f3620ceb4e4ee736bdbaf4ee7120ff4638d199720a89ad77ba673c4189235" Nov 01 21:58:07.393499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.389125 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" event={"ID":"787a27a9-8bce-4235-b2f3-f865827ecd90","Type":"ContainerDied","Data":"54efa380580e9302e35af185813aa4001a5d8e3ccef2790afeb2b585387031cb"} Nov 01 21:58:07.393499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.389160 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54efa380580e9302e35af185813aa4001a5d8e3ccef2790afeb2b585387031cb" Nov 01 21:58:07.393499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.389215 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79" Nov 01 21:58:07.398464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.397140 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:58:07.400431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.400411 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 21:58:07.402277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.402252 5036 generic.go:356] "Generic (PLEG): container finished" podID="f467320f-9bc7-47a4-962c-1769d84a1aae" containerID="dacce5b33d66817ecf0a49f86de0efa9a8a4b291b7642c156ab03c8550a9cb55" exitCode=0 Nov 01 21:58:07.403338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.403316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" event={"ID":"f467320f-9bc7-47a4-962c-1769d84a1aae","Type":"ContainerDied","Data":"dacce5b33d66817ecf0a49f86de0efa9a8a4b291b7642c156ab03c8550a9cb55"} Nov 01 21:58:07.432506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.430783 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:58:07.440795 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.440502 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 21:58:07.544543 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.544512 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\") pod \"660aaf4f-4ed4-4780-8571-9727560319bb\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " Nov 01 21:58:07.544990 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.544954 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\" (UniqueName: \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\") pod \"660aaf4f-4ed4-4780-8571-9727560319bb\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " Nov 01 21:58:07.545111 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.545095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\") pod \"660aaf4f-4ed4-4780-8571-9727560319bb\" (UID: \"660aaf4f-4ed4-4780-8571-9727560319bb\") " Nov 01 21:58:07.545326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.545295 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca" pod "660aaf4f-4ed4-4780-8571-9727560319bb" (UID: "660aaf4f-4ed4-4780-8571-9727560319bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:07.546391 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.546366 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:07.549125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.549105 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca" pod "660aaf4f-4ed4-4780-8571-9727560319bb" (UID: "660aaf4f-4ed4-4780-8571-9727560319bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:07.555006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.552388 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8" pod "660aaf4f-4ed4-4780-8571-9727560319bb" (UID: "660aaf4f-4ed4-4780-8571-9727560319bb"). InnerVolumeSpecName "kube-api-access-ngqd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:07.621622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.621289 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:58:07.622791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.622746 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 21:58:07.648893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.648864 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqd8\" (UniqueName: \"kubernetes.io/projected/660aaf4f-4ed4-4780-8571-9727560319bb-kube-api-access-ngqd8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:07.648893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:07.648893 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660aaf4f-4ed4-4780-8571-9727560319bb-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:08.129973 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.129931 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:08.410610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.410531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9" Nov 01 21:58:08.578074 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.578049 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:08.767258 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.767172 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\") pod \"f467320f-9bc7-47a4-962c-1769d84a1aae\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " Nov 01 21:58:08.767414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.767266 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\") pod \"f467320f-9bc7-47a4-962c-1769d84a1aae\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " Nov 01 21:58:08.767564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.767545 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\") pod \"f467320f-9bc7-47a4-962c-1769d84a1aae\" (UID: \"f467320f-9bc7-47a4-962c-1769d84a1aae\") " Nov 01 21:58:08.767926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.767889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca" pod "f467320f-9bc7-47a4-962c-1769d84a1aae" (UID: "f467320f-9bc7-47a4-962c-1769d84a1aae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:08.769656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.769624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627" pod "f467320f-9bc7-47a4-962c-1769d84a1aae" (UID: "f467320f-9bc7-47a4-962c-1769d84a1aae"). InnerVolumeSpecName "kube-api-access-cr627". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:08.769827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.769805 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca" pod "f467320f-9bc7-47a4-962c-1769d84a1aae" (UID: "f467320f-9bc7-47a4-962c-1769d84a1aae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:08.769915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.769847 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/f467320f-9bc7-47a4-962c-1769d84a1aae-kube-api-access-cr627\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:08.769915 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.769862 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:08.871267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.871234 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f467320f-9bc7-47a4-962c-1769d84a1aae-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:08.949319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:08.949255 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:09.420406 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:09.420378 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" Nov 01 21:58:09.420818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:09.420417 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n" event={"ID":"f467320f-9bc7-47a4-962c-1769d84a1aae","Type":"ContainerDied","Data":"f4bd541651333602317911161deb35620571275728fd6f528be272ac34ccd306"} Nov 01 21:58:09.420818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:09.420480 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4bd541651333602317911161deb35620571275728fd6f528be272ac34ccd306" Nov 01 21:58:09.639607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:09.639569 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:10.446496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:10.446424 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:11.234329 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:11.234286 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:12.212747 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:12.212711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:12.999581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:12.999540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:13.758833 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:13.758793 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:14.534819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:14.534775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:15.420579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.420539 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:15.788811 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.788764 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q"] Nov 01 21:58:15.789774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789750 5036 cpu_manager.go:414] "containerMap: removing container" podUID="787a27a9-8bce-4235-b2f3-f865827ecd90" containerName="revoke-keys" Nov 01 21:58:15.789857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789776 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787a27a9-8bce-4235-b2f3-f865827ecd90" containerName="revoke-keys" Nov 01 21:58:15.789857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789836 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f467320f-9bc7-47a4-962c-1769d84a1aae" containerName="revoke-keys" Nov 01 21:58:15.789857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789843 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f467320f-9bc7-47a4-962c-1769d84a1aae" containerName="revoke-keys" Nov 01 21:58:15.789857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789854 5036 cpu_manager.go:414] "containerMap: removing container" podUID="660aaf4f-4ed4-4780-8571-9727560319bb" containerName="revoke-keys" Nov 01 21:58:15.789998 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.789862 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660aaf4f-4ed4-4780-8571-9727560319bb" containerName="revoke-keys" Nov 01 21:58:15.790030 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.790011 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="787a27a9-8bce-4235-b2f3-f865827ecd90" containerName="revoke-keys" Nov 01 21:58:15.790069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.790030 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f467320f-9bc7-47a4-962c-1769d84a1aae" containerName="revoke-keys" Nov 01 21:58:15.790069 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.790041 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="660aaf4f-4ed4-4780-8571-9727560319bb" containerName="revoke-keys" Nov 01 21:58:15.795832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.795810 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.800725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.800669 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q"] Nov 01 21:58:15.801302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.801279 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:58:15.801476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.801429 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:58:15.801602 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.801510 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:58:15.803348 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.803309 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:58:15.883001 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.882910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lw5m\" (UniqueName: \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.883167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.883088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.883335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.883308 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.984848 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.984813 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.985005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.984931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lw5m\" (UniqueName: \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.985005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.984983 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.985911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.985883 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:15.987352 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:15.987322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:16.011725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.008210 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lw5m\" (UniqueName: \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\") pod \"maas-api-revoke-sub-9c93d125-mrd7q\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:16.146692 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.146620 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:16.327535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.327499 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w"] Nov 01 21:58:16.334042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.334014 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.334268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.334243 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w"] Nov 01 21:58:16.395700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.395669 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgrd\" (UniqueName: \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.395846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.395752 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.395846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.395821 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.445946 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:16.445919 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f983c2_966e_4ea4_9bef_93b99809a331.slice/crio-ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f WatchSource:0}: Error finding container ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f: Status 404 returned error can't find the container with id ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f Nov 01 21:58:16.447238 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.447212 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q"] Nov 01 21:58:16.483571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.483542 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:16.500386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.499862 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgrd\" (UniqueName: \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.500386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.500098 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.500386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.500199 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.502869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.502825 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.504615 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.504586 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" event={"ID":"21f983c2-966e-4ea4-9bef-93b99809a331","Type":"ContainerStarted","Data":"ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f"} Nov 01 21:58:16.577894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.577861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.578269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.578242 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgrd\" (UniqueName: \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\") pod \"maas-api-revoke-sub-f7173595-49f5w\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.653284 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.653256 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:16.835035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.835002 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx"] Nov 01 21:58:16.841121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.841101 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:16.853399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.852344 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx"] Nov 01 21:58:16.906660 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.906629 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:16.906785 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.906692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:16.906887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.906861 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5cq\" (UniqueName: \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:16.952923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:16.952823 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w"] Nov 01 21:58:17.008123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.008093 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.008250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.008169 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.008250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.008228 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5cq\" (UniqueName: \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.009258 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.009231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.010602 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.010581 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.028124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.028102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5cq\" (UniqueName: \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\") pod \"maas-api-revoke-sub-ec90963f-5mclx\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.160186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.160158 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:17.248524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.246779 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:17.453768 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:17.453731 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc5fa1_26e6_44bc_ad10_27c44261e527.slice/crio-4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715 WatchSource:0}: Error finding container 4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715: Status 404 returned error can't find the container with id 4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715 Nov 01 21:58:17.454887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.454863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx"] Nov 01 21:58:17.517262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.517233 5036 generic.go:356] "Generic (PLEG): container finished" podID="21f983c2-966e-4ea4-9bef-93b99809a331" containerID="7bab04eb59523a7419fed49e81e7e27f5458a91a80b8d10e36289b6e93416a91" exitCode=0 Nov 01 21:58:17.517401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.517323 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" event={"ID":"21f983c2-966e-4ea4-9bef-93b99809a331","Type":"ContainerDied","Data":"7bab04eb59523a7419fed49e81e7e27f5458a91a80b8d10e36289b6e93416a91"} Nov 01 21:58:17.518823 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.518797 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" event={"ID":"a9dc5fa1-26e6-44bc-ad10-27c44261e527","Type":"ContainerStarted","Data":"4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715"} Nov 01 21:58:17.520777 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.520757 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" event={"ID":"eea941d9-5123-4441-886b-138af23c6b5f","Type":"ContainerStarted","Data":"15624c63be1feddbcedceb8303e845f4a9c0fb98b4a76669a163cac1fa85de65"} Nov 01 21:58:17.520900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.520800 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" event={"ID":"eea941d9-5123-4441-886b-138af23c6b5f","Type":"ContainerStarted","Data":"6c07880bdc08c3324822d526046351c65df9a2118fb15a26ea434bd4ad8e54ba"} Nov 01 21:58:17.555594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.555549 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" podStartSLOduration=1.555537287 podStartE2EDuration="1.555537287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:17.553943666 +0000 UTC m=+851.246924152" watchObservedRunningTime="2026-11-01 21:58:17.555537287 +0000 UTC m=+851.248517798" Nov 01 21:58:17.851982 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.851941 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 21:58:17.859795 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.859749 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.865217 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.865194 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 21:58:17.870236 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.870217 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.872845 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.872818 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 21:58:17.895536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.895498 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.924235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.924287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.924373 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.924397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.924425 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925118 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925147 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925182 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlchb\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925210 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556d4\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925239 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.927483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.925340 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.928245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.927513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.928245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.927590 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.928245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.927634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:17.928245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.927694 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:17.946558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:17.946528 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:18.029081 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029039 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029107 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029143 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlchb\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556d4\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029250 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029370 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029393 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029483 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029547 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029625 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029783 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029703 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029782 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.029818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029816 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.029921 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.029865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.030368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.030339 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.031070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.030686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.031070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.030757 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.031070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.030994 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.031412 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.031389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.031882 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.031859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.032224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.032197 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.032530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.032507 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.032624 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.032567 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.032709 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.032687 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.034173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.034146 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.034872 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.034852 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.035501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.035471 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.035679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.035661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.044791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.044732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.045068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.045046 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556d4\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.048116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.045861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.048116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.046137 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlchb\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.182366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.182338 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:18.224356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.224108 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:18.238932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.238882 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 21:58:18.255916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.254542 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 21:58:18.255916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.254646 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335575 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335623 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335650 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335714 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335776 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335837 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.335906 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgh4\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.336372 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.336067 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437240 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437440 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437362 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437440 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437411 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437739 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437806 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437786 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437809 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.437917 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437898 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.438003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.437982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.438142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.438086 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgh4\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.438252 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.438229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.438779 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.438751 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.438892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.438861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.440530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.440503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.440642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.440616 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.451340 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.451312 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.451744 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.451725 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgh4\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.494301 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.494263 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 21:58:18.532832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.532800 5036 generic.go:356] "Generic (PLEG): container finished" podID="a9dc5fa1-26e6-44bc-ad10-27c44261e527" containerID="62547ce60a16d8c6538eb0d20b2e21c9e131a23fb8e270e5ab9597fa89b54b9f" exitCode=0 Nov 01 21:58:18.532930 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.532874 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" event={"ID":"a9dc5fa1-26e6-44bc-ad10-27c44261e527","Type":"ContainerDied","Data":"62547ce60a16d8c6538eb0d20b2e21c9e131a23fb8e270e5ab9597fa89b54b9f"} Nov 01 21:58:18.534533 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.534494 5036 generic.go:356] "Generic (PLEG): container finished" podID="eea941d9-5123-4441-886b-138af23c6b5f" containerID="15624c63be1feddbcedceb8303e845f4a9c0fb98b4a76669a163cac1fa85de65" exitCode=0 Nov 01 21:58:18.534632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.534569 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" event={"ID":"eea941d9-5123-4441-886b-138af23c6b5f","Type":"ContainerDied","Data":"15624c63be1feddbcedceb8303e845f4a9c0fb98b4a76669a163cac1fa85de65"} Nov 01 21:58:18.536405 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:18.536385 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed97d34_a363_477c_8113_57875f045680.slice/crio-ca52e0dacdea263674d4eebfab12d2a4f7eb9021ce127ba0f95bdae445068bef WatchSource:0}: Error finding container ca52e0dacdea263674d4eebfab12d2a4f7eb9021ce127ba0f95bdae445068bef: Status 404 returned error can't find the container with id ca52e0dacdea263674d4eebfab12d2a4f7eb9021ce127ba0f95bdae445068bef Nov 01 21:58:18.539380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.539350 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.539494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.539423 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.539494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.539486 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.581374 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.581342 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 21:58:18.584724 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:18.584679 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4906a4_cf6d_4462_be7a_c8802c4e282a.slice/crio-5ffdc9211fa15b8068dac5c7be888f87620d0795356a9da740c254f06c3e3ec4 WatchSource:0}: Error finding container 5ffdc9211fa15b8068dac5c7be888f87620d0795356a9da740c254f06c3e3ec4: Status 404 returned error can't find the container with id 5ffdc9211fa15b8068dac5c7be888f87620d0795356a9da740c254f06c3e3ec4 Nov 01 21:58:18.586031 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.586005 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:18.589786 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.589742 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.589907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.589819 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.589907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.589862 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.763486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.762578 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:18.852130 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.852042 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\") pod \"21f983c2-966e-4ea4-9bef-93b99809a331\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " Nov 01 21:58:18.852294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.852137 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\" (UniqueName: \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\") pod \"21f983c2-966e-4ea4-9bef-93b99809a331\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " Nov 01 21:58:18.852294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.852168 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\") pod \"21f983c2-966e-4ea4-9bef-93b99809a331\" (UID: \"21f983c2-966e-4ea4-9bef-93b99809a331\") " Nov 01 21:58:18.852423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.852393 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca" pod "21f983c2-966e-4ea4-9bef-93b99809a331" (UID: "21f983c2-966e-4ea4-9bef-93b99809a331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:18.853305 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.853264 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:18.873573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.873517 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca" pod "21f983c2-966e-4ea4-9bef-93b99809a331" (UID: "21f983c2-966e-4ea4-9bef-93b99809a331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:18.874977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.874932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m" pod "21f983c2-966e-4ea4-9bef-93b99809a331" (UID: "21f983c2-966e-4ea4-9bef-93b99809a331"). InnerVolumeSpecName "kube-api-access-9lw5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:18.900397 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:18.896762 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:58:18.921846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.921809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:18.955485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.955026 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lw5m\" (UniqueName: \"kubernetes.io/projected/21f983c2-966e-4ea4-9bef-93b99809a331-kube-api-access-9lw5m\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:18.955485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.955063 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f983c2-966e-4ea4-9bef-93b99809a331-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:18.962857 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.962817 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.963354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.963332 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.963574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.963555 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:58:18.978715 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:18.978115 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 21:58:19.547295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.547261 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" event={"ID":"0ed97d34-a363-477c-8113-57875f045680","Type":"ContainerStarted","Data":"a8efc8422551f279832b6e3d47060d5be95809267aa54cba405ab7deb282b2bc"} Nov 01 21:58:19.547826 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.547314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" event={"ID":"0ed97d34-a363-477c-8113-57875f045680","Type":"ContainerStarted","Data":"ca52e0dacdea263674d4eebfab12d2a4f7eb9021ce127ba0f95bdae445068bef"} Nov 01 21:58:19.549191 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.549163 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" Nov 01 21:58:19.549319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.549190 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q" event={"ID":"21f983c2-966e-4ea4-9bef-93b99809a331","Type":"ContainerDied","Data":"ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f"} Nov 01 21:58:19.549319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.549247 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed13d3406233b12c09d259b2bd352dd3d3c6e3d3105205662878bfea779b397f" Nov 01 21:58:19.554720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.554695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" event={"ID":"20c5c171-9539-4e97-994e-23751c85579f","Type":"ContainerStarted","Data":"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc"} Nov 01 21:58:19.554884 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.554866 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" event={"ID":"20c5c171-9539-4e97-994e-23751c85579f","Type":"ContainerStarted","Data":"62b1472c4e3ee34b36a813c40b51fb2ad861f96e433f2d4d826b6e2fc2b961e5"} Nov 01 21:58:19.558675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.558653 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" event={"ID":"cd4906a4-cf6d-4462-be7a-c8802c4e282a","Type":"ContainerStarted","Data":"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77"} Nov 01 21:58:19.558767 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.558695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" event={"ID":"cd4906a4-cf6d-4462-be7a-c8802c4e282a","Type":"ContainerStarted","Data":"5ffdc9211fa15b8068dac5c7be888f87620d0795356a9da740c254f06c3e3ec4"} Nov 01 21:58:19.639499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.632151 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:19.639499 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.638606 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podStartSLOduration=2.638585544 podStartE2EDuration="2.638585544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.573932675 +0000 UTC m=+853.266913154" watchObservedRunningTime="2026-11-01 21:58:19.638585544 +0000 UTC m=+853.331566033" Nov 01 21:58:19.670473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.669264 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podStartSLOduration=2.669250198 podStartE2EDuration="2.669250198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.66232724 +0000 UTC m=+853.355307728" watchObservedRunningTime="2026-11-01 21:58:19.669250198 +0000 UTC m=+853.362230735" Nov 01 21:58:19.716537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.711043 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:19.721581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.720391 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podStartSLOduration=1.720376272 podStartE2EDuration="1.720376272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.70340635 +0000 UTC m=+853.396386838" watchObservedRunningTime="2026-11-01 21:58:19.720376272 +0000 UTC m=+853.413356757" Nov 01 21:58:19.867923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.867844 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb"] Nov 01 21:58:19.868849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.868831 5036 cpu_manager.go:414] "containerMap: removing container" podUID="21f983c2-966e-4ea4-9bef-93b99809a331" containerName="revoke-keys" Nov 01 21:58:19.869546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.869528 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f983c2-966e-4ea4-9bef-93b99809a331" containerName="revoke-keys" Nov 01 21:58:19.869868 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.869851 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f983c2-966e-4ea4-9bef-93b99809a331" containerName="revoke-keys" Nov 01 21:58:19.973475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.968787 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:19.973475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:19.972852 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb"] Nov 01 21:58:20.058466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.058028 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.058466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.058088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm66\" (UniqueName: \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.058466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.058168 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.159850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.159757 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.160099 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.160081 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm66\" (UniqueName: \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.160262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.160249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.161709 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.161686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.167367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.167200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.243886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.235512 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:20.243886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.235810 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:20.244599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.244240 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:20.244599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.244292 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:20.281089 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.278683 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:20.281089 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.278747 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:20.320472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.312537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm66\" (UniqueName: \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\") pod \"maas-api-revoke-sub-2a536cbc-nkjhb\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.330471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.328003 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:20.486558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.485979 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\") pod \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " Nov 01 21:58:20.486558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.486108 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\" (UniqueName: \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\") pod \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " Nov 01 21:58:20.486558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.486159 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\") pod \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\" (UID: \"a9dc5fa1-26e6-44bc-ad10-27c44261e527\") " Nov 01 21:58:20.486828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.486684 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:20.640679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.567979 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca" pod "a9dc5fa1-26e6-44bc-ad10-27c44261e527" (UID: "a9dc5fa1-26e6-44bc-ad10-27c44261e527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:20.640679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.627703 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:20.640679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.628137 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:20.640679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.628186 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:20.780597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.734071 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:20.852472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.844001 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\") pod \"eea941d9-5123-4441-886b-138af23c6b5f\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " Nov 01 21:58:20.853015 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.852984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\") pod \"eea941d9-5123-4441-886b-138af23c6b5f\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " Nov 01 21:58:20.886620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.886584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\" (UniqueName: \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\") pod \"eea941d9-5123-4441-886b-138af23c6b5f\" (UID: \"eea941d9-5123-4441-886b-138af23c6b5f\") " Nov 01 21:58:20.890530 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.860850 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca" pod "eea941d9-5123-4441-886b-138af23c6b5f" (UID: "eea941d9-5123-4441-886b-138af23c6b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:21.016524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:20.997433 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:21.129221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.129101 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:21.130658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.130538 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:21.301473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.269725 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq" pod "a9dc5fa1-26e6-44bc-ad10-27c44261e527" (UID: "a9dc5fa1-26e6-44bc-ad10-27c44261e527"). InnerVolumeSpecName "kube-api-access-gv5cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:21.301473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.269871 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:21.301473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.269938 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:21.301792 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.301596 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca" pod "a9dc5fa1-26e6-44bc-ad10-27c44261e527" (UID: "a9dc5fa1-26e6-44bc-ad10-27c44261e527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:21.318467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.313613 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca" pod "eea941d9-5123-4441-886b-138af23c6b5f" (UID: "eea941d9-5123-4441-886b-138af23c6b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:21.318467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.313715 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:21.318467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.313773 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:21.318467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.315596 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd" pod "eea941d9-5123-4441-886b-138af23c6b5f" (UID: "eea941d9-5123-4441-886b-138af23c6b5f"). InnerVolumeSpecName "kube-api-access-rmgrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:21.368463 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.368413 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgrd\" (UniqueName: \"kubernetes.io/projected/eea941d9-5123-4441-886b-138af23c6b5f-kube-api-access-rmgrd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:21.368738 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.368714 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea941d9-5123-4441-886b-138af23c6b5f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:21.368849 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.368835 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5cq\" (UniqueName: \"kubernetes.io/projected/a9dc5fa1-26e6-44bc-ad10-27c44261e527-kube-api-access-gv5cq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:21.368937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.368926 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc5fa1-26e6-44bc-ad10-27c44261e527-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:21.444490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.441186 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" event={"ID":"eea941d9-5123-4441-886b-138af23c6b5f","Type":"ContainerDied","Data":"6c07880bdc08c3324822d526046351c65df9a2118fb15a26ea434bd4ad8e54ba"} Nov 01 21:58:21.444490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.441258 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c07880bdc08c3324822d526046351c65df9a2118fb15a26ea434bd4ad8e54ba" Nov 01 21:58:21.444490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.441336 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w" Nov 01 21:58:21.464472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.464050 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" Nov 01 21:58:21.469682 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.469643 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx" event={"ID":"a9dc5fa1-26e6-44bc-ad10-27c44261e527","Type":"ContainerDied","Data":"4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715"} Nov 01 21:58:21.471893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.471870 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a483b48d6bf98e0744604d66cc000c4a2c7211ebd1329f04063e03d80599715" Nov 01 21:58:21.671759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.671335 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:21.726634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.712422 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:21.726634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.712496 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:21.726634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.714231 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:21.726634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.714313 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:21.767470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:21.766176 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:22.079469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.077574 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:22.079469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.078131 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:22.134474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.124558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:22.167863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.144164 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:22.229620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.229577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:22.248471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.236536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:22.248471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.246158 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:58:22.255171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.255140 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:22.255400 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.255366 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:22.255514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.255422 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:22.255514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.255369 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:22.273731 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.273662 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:22.306610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.275076 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:22.333566 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.322676 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:22.752343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.752288 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:22.766237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.765607 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:22.847739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.847704 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:22.848002 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.847973 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:22.891772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:22.891688 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:23.057791 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.057760 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb"] Nov 01 21:58:23.078269 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.078240 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:58:23.078546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.078529 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:23.120464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.084833 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:23.120464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.118789 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" event={"ID":"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9","Type":"ContainerStarted","Data":"f64ffa4f1abcbe33f66ae3edbd19becf8a0660be9407bbbc1b926dee905f9c58"} Nov 01 21:58:23.120464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.118941 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:58:23.139432 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.139397 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:23.144322 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.144247 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:23.144792 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.144773 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:58:23.146134 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.146112 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:23.146832 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.146815 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:58:23.169862 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.169786 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:23.171010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.170982 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:58:23.216634 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.216597 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:23.216897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.216867 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:23.242112 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.242001 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:23.242112 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.242051 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:23.247742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.245951 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:23.256013 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.254916 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:23.256468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.256256 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:58:23.630479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.588365 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:23.708466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.707531 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:23.708466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.591479 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:23.817612 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.817577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:58:23.851886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:23.851840 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.152922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.145391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.152922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.147083 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:24.152922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.147387 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.152922 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.147898 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:58:24.165470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.164470 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.175561 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.175522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:24.297590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.295421 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" event={"ID":"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9","Type":"ContainerStarted","Data":"2998108e54f9515815a24b88c080823d3b8963f64181b9ca050ad2d56d05a682"} Nov 01 21:58:24.345297 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.333904 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:24.347827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.347769 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:24.379408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.371011 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:24.379408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.360215 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.379408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.355973 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:24.379408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.354894 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:24.379408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.371395 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:24.617846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.617810 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:24.618119 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.618090 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:24.647545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.647500 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.820579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.820530 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.827465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.827412 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:24.866551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:24.853516 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" podStartSLOduration=5.853492967 podStartE2EDuration="5.853492967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:24.360159033 +0000 UTC m=+858.053139520" watchObservedRunningTime="2026-11-01 21:58:24.853492967 +0000 UTC m=+858.546473470" Nov 01 21:58:25.025532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.021547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.146469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.145693 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.148278 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.147538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:25.182470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.165547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.183326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.183284 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.192472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.183831 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:25.192690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.192662 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:25.278060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.277964 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:25.278336 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.278302 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:25.347567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.347522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.348123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.348093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:25.356626 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.356589 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 01 21:58:25.365658 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.364822 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.371356 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.371315 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.384466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.376138 5036 generic.go:356] "Generic (PLEG): container finished" podID="7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" containerID="2998108e54f9515815a24b88c080823d3b8963f64181b9ca050ad2d56d05a682" exitCode=0 Nov 01 21:58:25.384466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.376310 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" event={"ID":"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9","Type":"ContainerDied","Data":"2998108e54f9515815a24b88c080823d3b8963f64181b9ca050ad2d56d05a682"} Nov 01 21:58:25.597590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.590302 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:25.598284 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.597839 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:25.654957 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.654908 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.655765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.655738 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:25.904521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.901974 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.931479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:25.909544 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": context deadline exceeded" Nov 01 21:58:26.042607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.042557 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.043609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.043584 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:58:26.155718 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.152563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.155718 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.153332 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.174473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.166785 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.193475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.183804 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:26.193475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.183859 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:26.193475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.184257 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.309503 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.280568 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:26.309503 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.280632 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:26.349782 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.349740 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:26.356475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.350707 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:58:26.382467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.356702 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.383126 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.383092 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:26.711479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.649786 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:26.711479 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:26.671656 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.241496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.179819 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:58:27.241496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.187670 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.328067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.328031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:27.353758 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.353716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.388515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.355131 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.388515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.355189 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:27.388515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.355911 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.388515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.355959 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:27.452338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.427399 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.452841 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.452760 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:27.662637 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.662594 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.696474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.678660 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.696474 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:27.678717 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:28.256468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.232146 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.326602 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.326559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:58:28.383601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.383551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.658671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.658619 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.916793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.916692 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:28.955472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:28.952728 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.144468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.143552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.232909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.232813 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.265468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.259902 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:29.299646 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.294687 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:58:29.303896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.294840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:29.349567 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.349524 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:29.408642 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.408592 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.467857 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:29.467812 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.827s" Nov 01 21:58:29.497335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.469896 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:58:29.561468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.540965 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.561767 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.561729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.579657 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.579710 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.579812 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.579837 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.580109 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.587469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.580140 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:29.747878 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.747789 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.748154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.748105 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:29.762623 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.762591 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.762887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:29.762842 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:30.148470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.141326 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7"] Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.149777 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a9dc5fa1-26e6-44bc-ad10-27c44261e527" containerName="revoke-keys" Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.166535 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9dc5fa1-26e6-44bc-ad10-27c44261e527" containerName="revoke-keys" Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.166639 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eea941d9-5123-4441-886b-138af23c6b5f" containerName="revoke-keys" Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.166649 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea941d9-5123-4441-886b-138af23c6b5f" containerName="revoke-keys" Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.166891 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea941d9-5123-4441-886b-138af23c6b5f" containerName="revoke-keys" Nov 01 21:58:30.171620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.166924 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9dc5fa1-26e6-44bc-ad10-27c44261e527" containerName="revoke-keys" Nov 01 21:58:30.377758 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.377717 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:30.392312 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.377894 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:58:30.392312 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.377930 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:30.392605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.378032 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:30.405303 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.378051 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:30.405303 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.378070 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:30.407999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.378093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:30.419594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.378128 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:30.419594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.408879 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:30.439285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.419884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:58:30.439285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.377678 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:30.439285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.424120 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7"] Nov 01 21:58:30.439285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.424235 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.441198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.441150 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:30.499576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.499538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:30.664243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.660553 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:30.664243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.660620 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:30.664243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.661059 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:30.664243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.661105 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:30.688467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.687910 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:30.688467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.687963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:30.796589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.795470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkqf\" (UniqueName: \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.796589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.795726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.796589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.795858 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.878983 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.878951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:58:30.957816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.923193 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.957816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.923298 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.957816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.923357 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkqf\" (UniqueName: \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.957816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.934486 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:30.980103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.975138 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.980103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.975890 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:30.985678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:30.984655 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:58:31.089927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.089897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkqf\" (UniqueName: \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\") pod \"maas-api-revoke-sub-9285a513-lgzr7\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:31.184274 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.184160 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:31.218547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.218524 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:58:31.226351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.226331 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:58:31.252619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.238795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:31.252619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.241701 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:58:31.252619 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.252043 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:58:31.264326 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.264252 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:31.274242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.274218 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:31.277546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.277516 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:31.278062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.278041 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:58:31.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.346833 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:31.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.346883 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:31.355390 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.355368 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:58:31.394694 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.385683 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:31.394694 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.386772 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.423205 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.427807 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.429515 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\" (UniqueName: \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\") pod \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.429713 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\") pod \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.429782 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\") pod \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\" (UID: \"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9\") " Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.430783 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca" pod "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" (UID: "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.431884 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.438671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca" pod "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" (UID: "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:31.442802 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.440621 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66" pod "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" (UID: "7f7bed0f-3fdd-483d-99c5-f47ebfc932b9"). InnerVolumeSpecName "kube-api-access-ttm66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:31.449342 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.449320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:31.451163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.451145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:31.456207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.456188 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 21:58:31.542571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.541366 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttm66\" (UniqueName: \"kubernetes.io/projected/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-kube-api-access-ttm66\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:31.542571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.541631 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:31.587329 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.586892 5036 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 21:58:31.587329 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.586952 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 21:58:31.834863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.834498 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7"] Nov 01 21:58:31.842119 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.842087 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:31.996468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.990503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" event={"ID":"7f7bed0f-3fdd-483d-99c5-f47ebfc932b9","Type":"ContainerDied","Data":"f64ffa4f1abcbe33f66ae3edbd19becf8a0660be9407bbbc1b926dee905f9c58"} Nov 01 21:58:31.996468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.990565 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64ffa4f1abcbe33f66ae3edbd19becf8a0660be9407bbbc1b926dee905f9c58" Nov 01 21:58:31.996468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:31.990655 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb" Nov 01 21:58:32.019469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.001787 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" event={"ID":"9666743e-59a5-4d31-83db-ea9831ccf2db","Type":"ContainerStarted","Data":"be898210a9093b4e137a732d74d45c50ac772f3cbe7e188fdcb89636da3fce11"} Nov 01 21:58:32.228024 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.227983 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 21:58:32.228285 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.228257 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 21:58:32.339830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.339789 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs"] Nov 01 21:58:32.341038 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.341013 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" containerName="revoke-keys" Nov 01 21:58:32.341155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.341039 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" containerName="revoke-keys" Nov 01 21:58:32.341233 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.341218 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" containerName="revoke-keys" Nov 01 21:58:32.366230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.366020 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.475685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.466672 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs"] Nov 01 21:58:32.475685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.475666 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc"] Nov 01 21:58:32.499706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.481725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.545874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.545844 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.546249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.546222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjn7j\" (UniqueName: \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.546594 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.546574 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.607486 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc"] Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652344 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652400 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb24\" (UniqueName: \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652496 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjn7j\" (UniqueName: \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652775 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.652847 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.662939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.658476 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.750867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.742995 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjn7j\" (UniqueName: \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.752195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.752155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:32.780558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.755391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.780558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.755509 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.780558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.755600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb24\" (UniqueName: \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.780558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.764664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.780558 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.764842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ee1dfe7-bwrgs\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:32.796749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.796067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.858502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.850203 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb24\" (UniqueName: \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\") pod \"maas-api-revoke-sub-5f26d954-rb8qc\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:32.963423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:32.963176 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:33.049457 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.031069 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5"] Nov 01 21:58:33.097082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.093325 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:33.146689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.146656 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5"] Nov 01 21:58:33.146840 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.146778 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.172279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.172245 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" event={"ID":"9666743e-59a5-4d31-83db-ea9831ccf2db","Type":"ContainerStarted","Data":"65acfe8c4aa576a0c52e4c5b4726c8b26e9f10064c1be33889ea229171ad2a08"} Nov 01 21:58:33.178488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.175201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrhl\" (UniqueName: \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.178488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.175438 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.178488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.175549 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.251681 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.251558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:33.278481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.277826 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.278481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.277993 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.278481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.278105 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrhl\" (UniqueName: \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.280079 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.280000 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.322928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.320102 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrhl\" (UniqueName: \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.339244 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.339214 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24587cf-h2np5\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.459037 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.459006 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:33.459495 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.459474 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:33.460676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.460656 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 21:58:33.484290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.484233 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" podStartSLOduration=5.484215579 podStartE2EDuration="5.484215579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:33.252977376 +0000 UTC m=+866.945957866" watchObservedRunningTime="2026-11-01 21:58:33.484215579 +0000 UTC m=+867.177196065" Nov 01 21:58:33.486517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.486496 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:33.514204 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:33.514132 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a3e3cf_2e6e_4ab6_aa0d_633c0063d7d7.slice/crio-e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248 WatchSource:0}: Error finding container e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248: Status 404 returned error can't find the container with id e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248 Nov 01 21:58:33.516744 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.515822 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs"] Nov 01 21:58:33.546576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.546413 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:33.575890 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:33.575865 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb4413e_e309_401b_b225_bc06f442eee5.slice/crio-2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3 WatchSource:0}: Error finding container 2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3: Status 404 returned error can't find the container with id 2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3 Nov 01 21:58:33.576355 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.576317 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc"] Nov 01 21:58:33.858772 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:33.858744 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e0e9ff_417a_4016_be1c_3921683d532c.slice/crio-2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485 WatchSource:0}: Error finding container 2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485: Status 404 returned error can't find the container with id 2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485 Nov 01 21:58:33.860437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:33.860408 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5"] Nov 01 21:58:34.214086 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.214048 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" event={"ID":"abb4413e-e309-401b-b225-bc06f442eee5","Type":"ContainerStarted","Data":"15db8c4b5cb3ddfbf5c9c008f388bcd8e3d6c783014d1a0659a0ae5e05c94ee5"} Nov 01 21:58:34.214645 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.214618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" event={"ID":"abb4413e-e309-401b-b225-bc06f442eee5","Type":"ContainerStarted","Data":"2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3"} Nov 01 21:58:34.224883 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.224852 5036 generic.go:356] "Generic (PLEG): container finished" podID="9666743e-59a5-4d31-83db-ea9831ccf2db" containerID="65acfe8c4aa576a0c52e4c5b4726c8b26e9f10064c1be33889ea229171ad2a08" exitCode=0 Nov 01 21:58:34.225166 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.225142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" event={"ID":"9666743e-59a5-4d31-83db-ea9831ccf2db","Type":"ContainerDied","Data":"65acfe8c4aa576a0c52e4c5b4726c8b26e9f10064c1be33889ea229171ad2a08"} Nov 01 21:58:34.227220 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.227196 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" event={"ID":"65e0e9ff-417a-4016-be1c-3921683d532c","Type":"ContainerStarted","Data":"2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485"} Nov 01 21:58:34.230881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.230791 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" event={"ID":"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7","Type":"ContainerStarted","Data":"339c33faec560dc83c27fe0cc89e6c8a13782fd85a1964e499ae7f274d727cb4"} Nov 01 21:58:34.230881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.230834 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" event={"ID":"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7","Type":"ContainerStarted","Data":"e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248"} Nov 01 21:58:34.248423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.248372 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" podStartSLOduration=2.248354795 podStartE2EDuration="2.248354795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:34.246802444 +0000 UTC m=+867.939782930" watchObservedRunningTime="2026-11-01 21:58:34.248354795 +0000 UTC m=+867.941335282" Nov 01 21:58:34.312957 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.312907 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:34.313128 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.312986 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:34.367514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.323226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:34.367514 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.347727 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" podStartSLOduration=2.347703271 podStartE2EDuration="2.347703271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:34.301422582 +0000 UTC m=+867.994403071" watchObservedRunningTime="2026-11-01 21:58:34.347703271 +0000 UTC m=+868.040683757" Nov 01 21:58:34.855172 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.855130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:34.911313 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:34.911211 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 21:58:35.068308 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.068274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:35.242120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.242088 5036 generic.go:356] "Generic (PLEG): container finished" podID="abb4413e-e309-401b-b225-bc06f442eee5" containerID="15db8c4b5cb3ddfbf5c9c008f388bcd8e3d6c783014d1a0659a0ae5e05c94ee5" exitCode=0 Nov 01 21:58:35.242315 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.242142 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" event={"ID":"abb4413e-e309-401b-b225-bc06f442eee5","Type":"ContainerDied","Data":"15db8c4b5cb3ddfbf5c9c008f388bcd8e3d6c783014d1a0659a0ae5e05c94ee5"} Nov 01 21:58:35.244724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.244563 5036 generic.go:356] "Generic (PLEG): container finished" podID="65e0e9ff-417a-4016-be1c-3921683d532c" containerID="10502d7dd4293bb9c51f8c13690efd37c7001046056f854437dd9cfcbe66c1bf" exitCode=0 Nov 01 21:58:35.244724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.244624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" event={"ID":"65e0e9ff-417a-4016-be1c-3921683d532c","Type":"ContainerDied","Data":"10502d7dd4293bb9c51f8c13690efd37c7001046056f854437dd9cfcbe66c1bf"} Nov 01 21:58:35.248011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.247986 5036 generic.go:356] "Generic (PLEG): container finished" podID="d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" containerID="339c33faec560dc83c27fe0cc89e6c8a13782fd85a1964e499ae7f274d727cb4" exitCode=0 Nov 01 21:58:35.248133 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.248015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" event={"ID":"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7","Type":"ContainerDied","Data":"339c33faec560dc83c27fe0cc89e6c8a13782fd85a1964e499ae7f274d727cb4"} Nov 01 21:58:35.448180 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.448115 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:35.639183 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.639146 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\") pod \"9666743e-59a5-4d31-83db-ea9831ccf2db\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " Nov 01 21:58:35.639328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.639307 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\") pod \"9666743e-59a5-4d31-83db-ea9831ccf2db\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " Nov 01 21:58:35.639380 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.639341 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\" (UniqueName: \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\") pod \"9666743e-59a5-4d31-83db-ea9831ccf2db\" (UID: \"9666743e-59a5-4d31-83db-ea9831ccf2db\") " Nov 01 21:58:35.639720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.639679 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca" pod "9666743e-59a5-4d31-83db-ea9831ccf2db" (UID: "9666743e-59a5-4d31-83db-ea9831ccf2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:35.640171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.640145 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:35.641328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.641298 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca" pod "9666743e-59a5-4d31-83db-ea9831ccf2db" (UID: "9666743e-59a5-4d31-83db-ea9831ccf2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:35.641416 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.641398 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf" pod "9666743e-59a5-4d31-83db-ea9831ccf2db" (UID: "9666743e-59a5-4d31-83db-ea9831ccf2db"). InnerVolumeSpecName "kube-api-access-fgkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:35.741412 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.741342 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgkqf\" (UniqueName: \"kubernetes.io/projected/9666743e-59a5-4d31-83db-ea9831ccf2db-kube-api-access-fgkqf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:35.741412 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:35.741371 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9666743e-59a5-4d31-83db-ea9831ccf2db-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.041827 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.041789 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:36.258273 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.258237 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" Nov 01 21:58:36.258436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.258274 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7" event={"ID":"9666743e-59a5-4d31-83db-ea9831ccf2db","Type":"ContainerDied","Data":"be898210a9093b4e137a732d74d45c50ac772f3cbe7e188fdcb89636da3fce11"} Nov 01 21:58:36.258436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.258328 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be898210a9093b4e137a732d74d45c50ac772f3cbe7e188fdcb89636da3fce11" Nov 01 21:58:36.475132 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.475101 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:36.559967 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.559938 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\") pod \"65e0e9ff-417a-4016-be1c-3921683d532c\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " Nov 01 21:58:36.560076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.559994 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\" (UniqueName: \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\") pod \"65e0e9ff-417a-4016-be1c-3921683d532c\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " Nov 01 21:58:36.560222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.560202 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\") pod \"65e0e9ff-417a-4016-be1c-3921683d532c\" (UID: \"65e0e9ff-417a-4016-be1c-3921683d532c\") " Nov 01 21:58:36.560478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.560435 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca" pod "65e0e9ff-417a-4016-be1c-3921683d532c" (UID: "65e0e9ff-417a-4016-be1c-3921683d532c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:36.560901 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.560878 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.562491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.562467 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca" pod "65e0e9ff-417a-4016-be1c-3921683d532c" (UID: "65e0e9ff-417a-4016-be1c-3921683d532c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:36.562655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.562639 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl" pod "65e0e9ff-417a-4016-be1c-3921683d532c" (UID: "65e0e9ff-417a-4016-be1c-3921683d532c"). InnerVolumeSpecName "kube-api-access-xlrhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:36.575437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.575415 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:36.582313 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.581705 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:36.661927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.661868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\" (UniqueName: \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\") pod \"abb4413e-e309-401b-b225-bc06f442eee5\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " Nov 01 21:58:36.662113 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.661989 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\") pod \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " Nov 01 21:58:36.662113 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.662022 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\" (UniqueName: \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\") pod \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " Nov 01 21:58:36.662113 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.662085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\") pod \"abb4413e-e309-401b-b225-bc06f442eee5\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " Nov 01 21:58:36.662298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.662116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\") pod \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\" (UID: \"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7\") " Nov 01 21:58:36.662359 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.662294 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\") pod \"abb4413e-e309-401b-b225-bc06f442eee5\" (UID: \"abb4413e-e309-401b-b225-bc06f442eee5\") " Nov 01 21:58:36.662918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.662887 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca" pod "abb4413e-e309-401b-b225-bc06f442eee5" (UID: "abb4413e-e309-401b-b225-bc06f442eee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:36.663070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.663051 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0e9ff-417a-4016-be1c-3921683d532c-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.663140 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.663078 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrhl\" (UniqueName: \"kubernetes.io/projected/65e0e9ff-417a-4016-be1c-3921683d532c-kube-api-access-xlrhl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.663140 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.663095 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.663430 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.663407 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca" pod "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" (UID: "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:36.664966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.664935 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca" pod "abb4413e-e309-401b-b225-bc06f442eee5" (UID: "abb4413e-e309-401b-b225-bc06f442eee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:36.665393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.665368 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24" pod "abb4413e-e309-401b-b225-bc06f442eee5" (UID: "abb4413e-e309-401b-b225-bc06f442eee5"). InnerVolumeSpecName "kube-api-access-7tb24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:36.665935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.665904 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j" pod "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" (UID: "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7"). InnerVolumeSpecName "kube-api-access-mjn7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:36.666664 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.666633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca" pod "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" (UID: "d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:36.764498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.764433 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tb24\" (UniqueName: \"kubernetes.io/projected/abb4413e-e309-401b-b225-bc06f442eee5-kube-api-access-7tb24\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.764498 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.764500 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.764685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.764510 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjn7j\" (UniqueName: \"kubernetes.io/projected/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-kube-api-access-mjn7j\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.764685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.764520 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb4413e-e309-401b-b225-bc06f442eee5-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.764685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:36.764529 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:37.060966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.060922 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:37.269551 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.269521 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" Nov 01 21:58:37.269706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.269516 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs" event={"ID":"d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7","Type":"ContainerDied","Data":"e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248"} Nov 01 21:58:37.269706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.269694 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67d371d028f6fcd4bbab4763bd1ec7a4ae1dbdf2b54d1e704827d92e243b248" Nov 01 21:58:37.271267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.271247 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" Nov 01 21:58:37.271409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.271297 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc" event={"ID":"abb4413e-e309-401b-b225-bc06f442eee5","Type":"ContainerDied","Data":"2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3"} Nov 01 21:58:37.271409 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.271359 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2fbf5e6505f1b06221e7f5daf38be49c363303a791548f68cbee055d5d32d3" Nov 01 21:58:37.273005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.272982 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" event={"ID":"65e0e9ff-417a-4016-be1c-3921683d532c","Type":"ContainerDied","Data":"2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485"} Nov 01 21:58:37.273101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.273009 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5" Nov 01 21:58:37.273101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:37.273015 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9e370e2ed14fad9829a6d4c4cae9ffcc5bba69387efc607c056b0292775485" Nov 01 21:58:38.042588 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:38.042554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:39.044975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:39.044938 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:39.856579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:39.856536 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:40.174146 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174074 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml"] Nov 01 21:58:40.174831 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174812 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" containerName="revoke-keys" Nov 01 21:58:40.174879 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174832 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" containerName="revoke-keys" Nov 01 21:58:40.174879 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174854 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9666743e-59a5-4d31-83db-ea9831ccf2db" containerName="revoke-keys" Nov 01 21:58:40.174879 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174859 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9666743e-59a5-4d31-83db-ea9831ccf2db" containerName="revoke-keys" Nov 01 21:58:40.174879 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174874 5036 cpu_manager.go:414] "containerMap: removing container" podUID="65e0e9ff-417a-4016-be1c-3921683d532c" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174879 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e0e9ff-417a-4016-be1c-3921683d532c" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174892 5036 cpu_manager.go:414] "containerMap: removing container" podUID="abb4413e-e309-401b-b225-bc06f442eee5" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.174898 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb4413e-e309-401b-b225-bc06f442eee5" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.175002 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.175014 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9666743e-59a5-4d31-83db-ea9831ccf2db" containerName="revoke-keys" Nov 01 21:58:40.175023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.175022 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e0e9ff-417a-4016-be1c-3921683d532c" containerName="revoke-keys" Nov 01 21:58:40.175198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.175028 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb4413e-e309-401b-b225-bc06f442eee5" containerName="revoke-keys" Nov 01 21:58:40.180029 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.180009 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.187896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.185920 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:58:40.187896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.186044 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:58:40.187896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.186075 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:58:40.187896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.186580 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:58:40.188994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.188969 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml"] Nov 01 21:58:40.213010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.212979 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.213163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.213124 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.213283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.213251 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtv4w\" (UniqueName: \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.314597 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.314567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.314787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.314632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.314787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.314697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtv4w\" (UniqueName: \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.315366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.315340 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.317228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.317204 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.327210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.327185 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtv4w\" (UniqueName: \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\") pod \"maas-api-revoke-sub-a49287ea-txnml\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.499057 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.498987 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:40.702028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.700915 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:40.791590 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:40.791552 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3881852_34f9_423b_9054_eb18ba2c78bc.slice/crio-54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635 WatchSource:0}: Error finding container 54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635: Status 404 returned error can't find the container with id 54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635 Nov 01 21:58:40.794248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:40.794221 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml"] Nov 01 21:58:41.318042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:41.318000 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" event={"ID":"f3881852-34f9-423b-9054-eb18ba2c78bc","Type":"ContainerStarted","Data":"8f2bf59994c8030ddb7ef9f67a2915a45e83d536a78b561f54c6ccf01d103f78"} Nov 01 21:58:41.318487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:41.318060 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" event={"ID":"f3881852-34f9-423b-9054-eb18ba2c78bc","Type":"ContainerStarted","Data":"54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635"} Nov 01 21:58:41.339889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:41.339803 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" podStartSLOduration=1.339784509 podStartE2EDuration="1.339784509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:41.336361308 +0000 UTC m=+875.029341796" watchObservedRunningTime="2026-11-01 21:58:41.339784509 +0000 UTC m=+875.032764996" Nov 01 21:58:41.429436 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:41.429398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:42.221019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.220987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:42.333179 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.333143 5036 generic.go:356] "Generic (PLEG): container finished" podID="f3881852-34f9-423b-9054-eb18ba2c78bc" containerID="8f2bf59994c8030ddb7ef9f67a2915a45e83d536a78b561f54c6ccf01d103f78" exitCode=0 Nov 01 21:58:42.333726 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.333228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" event={"ID":"f3881852-34f9-423b-9054-eb18ba2c78bc","Type":"ContainerDied","Data":"8f2bf59994c8030ddb7ef9f67a2915a45e83d536a78b561f54c6ccf01d103f78"} Nov 01 21:58:42.822102 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.822070 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4"] Nov 01 21:58:42.827011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.826991 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:42.839264 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.839236 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4"] Nov 01 21:58:42.952120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.952087 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:42.952120 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.952125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:42.952320 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:42.952215 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk4v\" (UniqueName: \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.053914 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.053883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.054082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.053935 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.054082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.053976 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk4v\" (UniqueName: \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.054743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.054715 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.056286 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.056259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.066299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.066278 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk4v\" (UniqueName: \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\") pod \"maas-api-revoke-sub-d186c4bd-zg6n4\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.117015 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.116932 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:43.146125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.146094 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:43.272354 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.271723 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm"] Nov 01 21:58:43.290083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.290054 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm"] Nov 01 21:58:43.290211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.290172 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.360413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.360327 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.361367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.360463 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.361367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.360494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.463121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.463084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.463319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.463177 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.463319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.463206 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.464988 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.464956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.477495 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.476421 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.485288 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.485189 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4"] Nov 01 21:58:43.494612 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.494575 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\") pod \"maas-api-revoke-sub-cfa14e88-j5ltm\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.522246 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.522226 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:43.564331 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.564302 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\" (UniqueName: \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\") pod \"f3881852-34f9-423b-9054-eb18ba2c78bc\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " Nov 01 21:58:43.564571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.564396 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\") pod \"f3881852-34f9-423b-9054-eb18ba2c78bc\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " Nov 01 21:58:43.564571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.564501 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\") pod \"f3881852-34f9-423b-9054-eb18ba2c78bc\" (UID: \"f3881852-34f9-423b-9054-eb18ba2c78bc\") " Nov 01 21:58:43.565298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.565254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca" pod "f3881852-34f9-423b-9054-eb18ba2c78bc" (UID: "f3881852-34f9-423b-9054-eb18ba2c78bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:43.566945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.566920 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca" pod "f3881852-34f9-423b-9054-eb18ba2c78bc" (UID: "f3881852-34f9-423b-9054-eb18ba2c78bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:43.567036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.566944 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w" pod "f3881852-34f9-423b-9054-eb18ba2c78bc" (UID: "f3881852-34f9-423b-9054-eb18ba2c78bc"). InnerVolumeSpecName "kube-api-access-dtv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:43.622192 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.622161 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:43.666978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.666951 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtv4w\" (UniqueName: \"kubernetes.io/projected/f3881852-34f9-423b-9054-eb18ba2c78bc-kube-api-access-dtv4w\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:43.666978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.666980 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:43.667146 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.666990 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3881852-34f9-423b-9054-eb18ba2c78bc-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:43.925679 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:43.925646 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm"] Nov 01 21:58:44.072595 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.072432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:44.368999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.368966 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" event={"ID":"d0c8ac64-a338-4cff-930c-1fcf33cd192d","Type":"ContainerStarted","Data":"ce4911021d4b82c7a3bc0a0e2578390a2f86289c412ae6601e6a7b817e49a644"} Nov 01 21:58:44.369396 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.369028 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" event={"ID":"d0c8ac64-a338-4cff-930c-1fcf33cd192d","Type":"ContainerStarted","Data":"7f379167914e5e490b48b5d74fa02ece38b723a7925d075a06f5a958da19c727"} Nov 01 21:58:44.370334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.370315 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" Nov 01 21:58:44.370418 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.370358 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml" event={"ID":"f3881852-34f9-423b-9054-eb18ba2c78bc","Type":"ContainerDied","Data":"54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635"} Nov 01 21:58:44.370418 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.370412 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54714a899d7508da5351992aac245e73d957aacd16897638800a046894671635" Nov 01 21:58:44.372293 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.372267 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" event={"ID":"0ff04bab-ff86-4df4-989b-9357fbbccbde","Type":"ContainerStarted","Data":"e30cebe9d24e404bd26bf03a795993b2c4b5015a44b721063ceed581834878d2"} Nov 01 21:58:44.372403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.372316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" event={"ID":"0ff04bab-ff86-4df4-989b-9357fbbccbde","Type":"ContainerStarted","Data":"60f11213adfa950f40dfb495c1384cd034fe56abf9cd55d3e0807f5ab140323f"} Nov 01 21:58:44.395638 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.395589 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" podStartSLOduration=1.395576364 podStartE2EDuration="1.395576364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:44.388295557 +0000 UTC m=+878.081276046" watchObservedRunningTime="2026-11-01 21:58:44.395576364 +0000 UTC m=+878.088556849" Nov 01 21:58:44.412621 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.412569 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" podStartSLOduration=2.412552057 podStartE2EDuration="2.412552057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:44.409725654 +0000 UTC m=+878.102706141" watchObservedRunningTime="2026-11-01 21:58:44.412552057 +0000 UTC m=+878.105532544" Nov 01 21:58:44.834687 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:44.834650 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:45.382676 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:45.382640 5036 generic.go:356] "Generic (PLEG): container finished" podID="0ff04bab-ff86-4df4-989b-9357fbbccbde" containerID="e30cebe9d24e404bd26bf03a795993b2c4b5015a44b721063ceed581834878d2" exitCode=0 Nov 01 21:58:45.383149 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:45.382727 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" event={"ID":"0ff04bab-ff86-4df4-989b-9357fbbccbde","Type":"ContainerDied","Data":"e30cebe9d24e404bd26bf03a795993b2c4b5015a44b721063ceed581834878d2"} Nov 01 21:58:45.384986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:45.384962 5036 generic.go:356] "Generic (PLEG): container finished" podID="d0c8ac64-a338-4cff-930c-1fcf33cd192d" containerID="ce4911021d4b82c7a3bc0a0e2578390a2f86289c412ae6601e6a7b817e49a644" exitCode=0 Nov 01 21:58:45.385106 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:45.385045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" event={"ID":"d0c8ac64-a338-4cff-930c-1fcf33cd192d","Type":"ContainerDied","Data":"ce4911021d4b82c7a3bc0a0e2578390a2f86289c412ae6601e6a7b817e49a644"} Nov 01 21:58:45.539248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:45.539214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:46.348335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.348301 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6944-f1e3c7"] Nov 01 21:58:46.349093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.349071 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f3881852-34f9-423b-9054-eb18ba2c78bc" containerName="revoke-keys" Nov 01 21:58:46.349198 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.349094 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3881852-34f9-423b-9054-eb18ba2c78bc" containerName="revoke-keys" Nov 01 21:58:46.349270 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.349241 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3881852-34f9-423b-9054-eb18ba2c78bc" containerName="revoke-keys" Nov 01 21:58:46.353515 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.353494 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:46.357812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.357211 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 21:58:46.360253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.358549 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6944-f1e3c7"] Nov 01 21:58:46.378609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.378295 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:46.528662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.528628 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwm2q\" (UniqueName: \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\") pod \"test-curl-6944-f1e3c7\" (UID: \"3385685f-48dd-41cf-b011-705b4e04689e\") " pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:46.630405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.630325 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwm2q\" (UniqueName: \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\") pod \"test-curl-6944-f1e3c7\" (UID: \"3385685f-48dd-41cf-b011-705b4e04689e\") " pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:46.645131 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.645105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwm2q\" (UniqueName: \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\") pod \"test-curl-6944-f1e3c7\" (UID: \"3385685f-48dd-41cf-b011-705b4e04689e\") " pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:46.708167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.703687 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4"] Nov 01 21:58:46.710040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.709714 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:46.710647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.710628 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.718613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.718589 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4"] Nov 01 21:58:46.720719 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.720700 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:46.752970 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.752792 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835293 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\") pod \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835406 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\") pod \"0ff04bab-ff86-4df4-989b-9357fbbccbde\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\") pod \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835614 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\" (UniqueName: \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\") pod \"0ff04bab-ff86-4df4-989b-9357fbbccbde\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835655 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\") pod \"0ff04bab-ff86-4df4-989b-9357fbbccbde\" (UID: \"0ff04bab-ff86-4df4-989b-9357fbbccbde\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.835697 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\") pod \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\" (UID: \"d0c8ac64-a338-4cff-930c-1fcf33cd192d\") " Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.836320 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9gt\" (UniqueName: \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.836366 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.836873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.836399 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.838173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.837512 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca" pod "d0c8ac64-a338-4cff-930c-1fcf33cd192d" (UID: "d0c8ac64-a338-4cff-930c-1fcf33cd192d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:46.838173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.837897 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca" pod "0ff04bab-ff86-4df4-989b-9357fbbccbde" (UID: "0ff04bab-ff86-4df4-989b-9357fbbccbde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:46.840659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.840620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v" pod "0ff04bab-ff86-4df4-989b-9357fbbccbde" (UID: "0ff04bab-ff86-4df4-989b-9357fbbccbde"). InnerVolumeSpecName "kube-api-access-rqk4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:46.840987 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.840952 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca" pod "0ff04bab-ff86-4df4-989b-9357fbbccbde" (UID: "0ff04bab-ff86-4df4-989b-9357fbbccbde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:46.841347 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.841312 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca" pod "d0c8ac64-a338-4cff-930c-1fcf33cd192d" (UID: "d0c8ac64-a338-4cff-930c-1fcf33cd192d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:46.844532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.844492 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d" pod "d0c8ac64-a338-4cff-930c-1fcf33cd192d" (UID: "d0c8ac64-a338-4cff-930c-1fcf33cd192d"). InnerVolumeSpecName "kube-api-access-qpm5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:46.938062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.937983 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.938216 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938203 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9gt\" (UniqueName: \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.938261 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.938300 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938272 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938300 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938292 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/d0c8ac64-a338-4cff-930c-1fcf33cd192d-kube-api-access-qpm5d\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938303 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk4v\" (UniqueName: \"kubernetes.io/projected/0ff04bab-ff86-4df4-989b-9357fbbccbde-kube-api-access-rqk4v\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938315 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff04bab-ff86-4df4-989b-9357fbbccbde-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938324 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938368 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938333 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8ac64-a338-4cff-930c-1fcf33cd192d-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:46.938749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.938729 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.940328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.940310 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:46.953123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:46.953101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9gt\" (UniqueName: \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\") pod \"maas-api-revoke-sub-a2d9cc98-b4xg4\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:47.037746 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.037718 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:47.058057 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:47.058029 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3385685f_48dd_41cf_b011_705b4e04689e.slice/crio-4efe9135bb0ec65268b52bd7cde99d052db629bca3ab9c251618479810efb985 WatchSource:0}: Error finding container 4efe9135bb0ec65268b52bd7cde99d052db629bca3ab9c251618479810efb985: Status 404 returned error can't find the container with id 4efe9135bb0ec65268b52bd7cde99d052db629bca3ab9c251618479810efb985 Nov 01 21:58:47.060535 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.060512 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6944-f1e3c7"] Nov 01 21:58:47.089360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.089324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:47.327680 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:47.327651 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5218e058_1d92_4f50_9327_711436d97642.slice/crio-f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487 WatchSource:0}: Error finding container f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487: Status 404 returned error can't find the container with id f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487 Nov 01 21:58:47.331909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.331878 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4"] Nov 01 21:58:47.467042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.467005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" event={"ID":"d0c8ac64-a338-4cff-930c-1fcf33cd192d","Type":"ContainerDied","Data":"7f379167914e5e490b48b5d74fa02ece38b723a7925d075a06f5a958da19c727"} Nov 01 21:58:47.467210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.467054 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f379167914e5e490b48b5d74fa02ece38b723a7925d075a06f5a958da19c727" Nov 01 21:58:47.467210 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.467019 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm" Nov 01 21:58:47.468516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.468492 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6944-f1e3c7" event={"ID":"3385685f-48dd-41cf-b011-705b4e04689e","Type":"ContainerStarted","Data":"4efe9135bb0ec65268b52bd7cde99d052db629bca3ab9c251618479810efb985"} Nov 01 21:58:47.469978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.469957 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" Nov 01 21:58:47.470048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.469977 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4" event={"ID":"0ff04bab-ff86-4df4-989b-9357fbbccbde","Type":"ContainerDied","Data":"60f11213adfa950f40dfb495c1384cd034fe56abf9cd55d3e0807f5ab140323f"} Nov 01 21:58:47.470048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.470024 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f11213adfa950f40dfb495c1384cd034fe56abf9cd55d3e0807f5ab140323f" Nov 01 21:58:47.471423 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.471397 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" event={"ID":"5218e058-1d92-4f50-9327-711436d97642","Type":"ContainerStarted","Data":"f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487"} Nov 01 21:58:47.893772 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:47.893281 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:48.484270 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:48.484236 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" event={"ID":"5218e058-1d92-4f50-9327-711436d97642","Type":"ContainerStarted","Data":"e83084dbd4fef08eb75866545e312bb9ec256341819f9952edecaa9cad845a0b"} Nov 01 21:58:48.504518 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:48.504424 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" podStartSLOduration=2.504410386 podStartE2EDuration="2.504410386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:48.502383024 +0000 UTC m=+882.195363511" watchObservedRunningTime="2026-11-01 21:58:48.504410386 +0000 UTC m=+882.197390871" Nov 01 21:58:48.537309 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:48.537279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:48.859741 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:48.859708 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:58:49.235685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:49.235594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:49.499846 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:49.499762 5036 generic.go:356] "Generic (PLEG): container finished" podID="5218e058-1d92-4f50-9327-711436d97642" containerID="e83084dbd4fef08eb75866545e312bb9ec256341819f9952edecaa9cad845a0b" exitCode=0 Nov 01 21:58:49.500003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:49.499859 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" event={"ID":"5218e058-1d92-4f50-9327-711436d97642","Type":"ContainerDied","Data":"e83084dbd4fef08eb75866545e312bb9ec256341819f9952edecaa9cad845a0b"} Nov 01 21:58:49.502167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:49.502139 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6944-f1e3c7" event={"ID":"3385685f-48dd-41cf-b011-705b4e04689e","Type":"ContainerStarted","Data":"74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40"} Nov 01 21:58:49.539330 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:49.539282 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6944-f1e3c7" podStartSLOduration=1.6324189690000002 podStartE2EDuration="3.539268203s" totalImagesPullingTime="1.906849234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:49.535678303 +0000 UTC m=+883.228658789" watchObservedRunningTime="2026-11-01 21:58:49.539268203 +0000 UTC m=+883.232248688" Nov 01 21:58:50.239207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.239172 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6944-f1e3c7"] Nov 01 21:58:50.246422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.246392 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6944-f1e3c7"] Nov 01 21:58:50.292599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.292564 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc"] Nov 01 21:58:50.293622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293595 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0ff04bab-ff86-4df4-989b-9357fbbccbde" containerName="revoke-keys" Nov 01 21:58:50.293622 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293621 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff04bab-ff86-4df4-989b-9357fbbccbde" containerName="revoke-keys" Nov 01 21:58:50.293819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293656 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d0c8ac64-a338-4cff-930c-1fcf33cd192d" containerName="revoke-keys" Nov 01 21:58:50.293819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293665 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c8ac64-a338-4cff-930c-1fcf33cd192d" containerName="revoke-keys" Nov 01 21:58:50.293819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293696 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3385685f-48dd-41cf-b011-705b4e04689e" containerName="test-curl-6944-f1e3c7" Nov 01 21:58:50.293819 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293703 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3385685f-48dd-41cf-b011-705b4e04689e" containerName="test-curl-6944-f1e3c7" Nov 01 21:58:50.294319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293877 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c8ac64-a338-4cff-930c-1fcf33cd192d" containerName="revoke-keys" Nov 01 21:58:50.294319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293895 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3385685f-48dd-41cf-b011-705b4e04689e" containerName="test-curl-6944-f1e3c7" Nov 01 21:58:50.294319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.293915 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff04bab-ff86-4df4-989b-9357fbbccbde" containerName="revoke-keys" Nov 01 21:58:50.304463 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.304430 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.308759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.307126 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc"] Nov 01 21:58:50.397892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.397853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvqx\" (UniqueName: \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.398047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.397966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.398179 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.398161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.398712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.398689 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6944-8f4772"] Nov 01 21:58:50.405398 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.405357 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:50.416168 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.414282 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6944-8f4772"] Nov 01 21:58:50.499997 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.499926 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvqx\" (UniqueName: \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.499997 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.499976 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.500207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.500064 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qbc\" (UniqueName: \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\") pod \"test-curl-6944-8f4772\" (UID: \"bbd8a474-9e5c-4aec-936a-b8ca8df24512\") " pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:50.500207 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.500130 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.500810 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.500785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.502697 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.502672 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.529283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.529253 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvqx\" (UniqueName: \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\") pod \"maas-api-revoke-sub-536f6e79-v4kjc\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.604055 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.603992 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qbc\" (UniqueName: \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\") pod \"test-curl-6944-8f4772\" (UID: \"bbd8a474-9e5c-4aec-936a-b8ca8df24512\") " pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:50.621490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.620672 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qbc\" (UniqueName: \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\") pod \"test-curl-6944-8f4772\" (UID: \"bbd8a474-9e5c-4aec-936a-b8ca8df24512\") " pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:50.625932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.625755 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:50.689221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.689173 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:50.730613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.729657 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:50.825529 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.809189 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\" (UniqueName: \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\") pod \"5218e058-1d92-4f50-9327-711436d97642\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " Nov 01 21:58:50.825529 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.809248 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\") pod \"5218e058-1d92-4f50-9327-711436d97642\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " Nov 01 21:58:50.825529 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.809330 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\") pod \"5218e058-1d92-4f50-9327-711436d97642\" (UID: \"5218e058-1d92-4f50-9327-711436d97642\") " Nov 01 21:58:50.828528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.828013 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca" pod "5218e058-1d92-4f50-9327-711436d97642" (UID: "5218e058-1d92-4f50-9327-711436d97642"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:50.830277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.830247 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca" pod "5218e058-1d92-4f50-9327-711436d97642" (UID: "5218e058-1d92-4f50-9327-711436d97642"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:50.843684 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.843640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt" pod "5218e058-1d92-4f50-9327-711436d97642" (UID: "5218e058-1d92-4f50-9327-711436d97642"). InnerVolumeSpecName "kube-api-access-8j9gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:50.928556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.928525 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5218e058-1d92-4f50-9327-711436d97642-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:50.928556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.928555 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9gt\" (UniqueName: \"kubernetes.io/projected/5218e058-1d92-4f50-9327-711436d97642-kube-api-access-8j9gt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:50.928556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.928572 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5218e058-1d92-4f50-9327-711436d97642-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:50.934040 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:50.934010 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0a394e_e5f5_4b0b_9a4c_6b07f0a8153e.slice/crio-9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771 WatchSource:0}: Error finding container 9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771: Status 404 returned error can't find the container with id 9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771 Nov 01 21:58:50.937796 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:50.937770 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc"] Nov 01 21:58:51.079342 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:51.079310 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd8a474_9e5c_4aec_936a_b8ca8df24512.slice/crio-b6aaf1d71a053f7192fcb81dc292e30845ceec407ca957e29728fc6d336b893b WatchSource:0}: Error finding container b6aaf1d71a053f7192fcb81dc292e30845ceec407ca957e29728fc6d336b893b: Status 404 returned error can't find the container with id b6aaf1d71a053f7192fcb81dc292e30845ceec407ca957e29728fc6d336b893b Nov 01 21:58:51.081455 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.081419 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6944-8f4772"] Nov 01 21:58:51.495083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.494999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:51.528975 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.528948 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" Nov 01 21:58:51.529109 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.528983 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4" event={"ID":"5218e058-1d92-4f50-9327-711436d97642","Type":"ContainerDied","Data":"f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487"} Nov 01 21:58:51.529109 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.529029 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b8d4d19dcc8e329eabcddfe60ba4ade560d6be696c72391f02deeb5e123487" Nov 01 21:58:51.530970 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.530940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6944-8f4772" event={"ID":"bbd8a474-9e5c-4aec-936a-b8ca8df24512","Type":"ContainerStarted","Data":"8e65461a2f253219916aa00fb25634e69259a31dd4e4fd71e5f94b569e115280"} Nov 01 21:58:51.531084 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.530983 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6944-8f4772" event={"ID":"bbd8a474-9e5c-4aec-936a-b8ca8df24512","Type":"ContainerStarted","Data":"b6aaf1d71a053f7192fcb81dc292e30845ceec407ca957e29728fc6d336b893b"} Nov 01 21:58:51.534439 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.534400 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6944-f1e3c7" podUID="3385685f-48dd-41cf-b011-705b4e04689e" containerName="test-curl-6944-f1e3c7" containerID="cri-o://74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40" gracePeriod=2 Nov 01 21:58:51.535752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.534910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" event={"ID":"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e","Type":"ContainerStarted","Data":"59536e7e9e10a28c90008754519b3c44bd2a24ee805e7e54c298e407b9c2395e"} Nov 01 21:58:51.535752 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.534964 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" event={"ID":"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e","Type":"ContainerStarted","Data":"9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771"} Nov 01 21:58:51.555373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.555323 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6944-8f4772" podStartSLOduration=1.555307223 podStartE2EDuration="1.555307223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:51.549115568 +0000 UTC m=+885.242096054" watchObservedRunningTime="2026-11-01 21:58:51.555307223 +0000 UTC m=+885.248287710" Nov 01 21:58:51.569999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:51.569936 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" podStartSLOduration=1.569919964 podStartE2EDuration="1.569919964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:51.567874178 +0000 UTC m=+885.260854706" watchObservedRunningTime="2026-11-01 21:58:51.569919964 +0000 UTC m=+885.262900450" Nov 01 21:58:52.228071 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:52.227994 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6944-8f4772"] Nov 01 21:58:52.235373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:52.235344 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6944-8f4772"] Nov 01 21:58:52.238630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:52.238576 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:52.544889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:52.544852 5036 generic.go:356] "Generic (PLEG): container finished" podID="1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" containerID="59536e7e9e10a28c90008754519b3c44bd2a24ee805e7e54c298e407b9c2395e" exitCode=0 Nov 01 21:58:52.545373 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:52.544929 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" event={"ID":"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e","Type":"ContainerDied","Data":"59536e7e9e10a28c90008754519b3c44bd2a24ee805e7e54c298e407b9c2395e"} Nov 01 21:58:53.031899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.031870 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4"] Nov 01 21:58:53.032591 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032575 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" containerName="test-curl-6944-8f4772" Nov 01 21:58:53.032643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032593 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" containerName="test-curl-6944-8f4772" Nov 01 21:58:53.032643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032636 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5218e058-1d92-4f50-9327-711436d97642" containerName="revoke-keys" Nov 01 21:58:53.032643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032643 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5218e058-1d92-4f50-9327-711436d97642" containerName="revoke-keys" Nov 01 21:58:53.032755 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032746 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" containerName="test-curl-6944-8f4772" Nov 01 21:58:53.032787 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.032764 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5218e058-1d92-4f50-9327-711436d97642" containerName="revoke-keys" Nov 01 21:58:53.036912 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.036891 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.046006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.045981 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4"] Nov 01 21:58:53.062064 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.062036 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.062226 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.062117 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qn6d\" (UniqueName: \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.062226 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.062207 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.126993 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.126962 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:53.163221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.163183 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.163481 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.163459 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.163574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.163551 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qn6d\" (UniqueName: \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.164196 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.164169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.177002 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.176968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.189175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.189143 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qn6d\" (UniqueName: \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\") pod \"maas-api-revoke-sub-6a0139ed-knds4\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.255042 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.255008 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j"] Nov 01 21:58:53.261023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.260999 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.267379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.267353 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j"] Nov 01 21:58:53.364743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.364658 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:58:53.366835 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.366809 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.366977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.366959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.367019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.366997 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kcc\" (UniqueName: \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.469019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.468982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.469553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.469166 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.469553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.469219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kcc\" (UniqueName: \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.469912 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.469886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.471478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.471423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.482337 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.482288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kcc\" (UniqueName: \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\") pod \"maas-api-revoke-sub-42189ac1-r5n6j\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.554078 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.554047 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6944-8f4772" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" containerName="test-curl-6944-8f4772" containerID="cri-o://8e65461a2f253219916aa00fb25634e69259a31dd4e4fd71e5f94b569e115280" gracePeriod=2 Nov 01 21:58:53.581571 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.581546 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:58:53.674365 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.674285 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4"] Nov 01 21:58:53.692791 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:53.692754 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feb9d45_8108_4a2f_af10_eaa3a10531ec.slice/crio-3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a WatchSource:0}: Error finding container 3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a: Status 404 returned error can't find the container with id 3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a Nov 01 21:58:53.740259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.739948 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:53.778818 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.778779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\") pod \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " Nov 01 21:58:53.779276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.779254 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\" (UniqueName: \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\") pod \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " Nov 01 21:58:53.779563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.779543 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\") pod \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\" (UID: \"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e\") " Nov 01 21:58:53.780054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.780030 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca" pod "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" (UID: "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:53.781422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.781391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca" pod "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" (UID: "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:53.781814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.781793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx" pod "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" (UID: "1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e"). InnerVolumeSpecName "kube-api-access-fwvqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:53.781970 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.781951 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:53.881190 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.881120 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\" (UniqueName: \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\") pod \"3385685f-48dd-41cf-b011-705b4e04689e\" (UID: \"3385685f-48dd-41cf-b011-705b4e04689e\") " Nov 01 21:58:53.882250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.881822 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:53.882250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.881853 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvqx\" (UniqueName: \"kubernetes.io/projected/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-kube-api-access-fwvqx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:53.882250 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.881870 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:53.884541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.884510 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q" pod "3385685f-48dd-41cf-b011-705b4e04689e" (UID: "3385685f-48dd-41cf-b011-705b4e04689e"). InnerVolumeSpecName "kube-api-access-pwm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:53.906309 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:58:53.906279 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b67b27_c12c_49ae_aa4c_b043da0c8e15.slice/crio-6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c WatchSource:0}: Error finding container 6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c: Status 404 returned error can't find the container with id 6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c Nov 01 21:58:53.908889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.908865 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j"] Nov 01 21:58:53.969232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.969201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:53.983419 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:53.983397 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwm2q\" (UniqueName: \"kubernetes.io/projected/3385685f-48dd-41cf-b011-705b4e04689e-kube-api-access-pwm2q\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:54.544017 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.543971 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3385685f-48dd-41cf-b011-705b4e04689e" path="/var/lib/kubelet/pods/3385685f-48dd-41cf-b011-705b4e04689e/volumes" Nov 01 21:58:54.604899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.604850 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" event={"ID":"9feb9d45-8108-4a2f-af10-eaa3a10531ec","Type":"ContainerStarted","Data":"a472bb2e7ade8eebacb2510486f7a0ae3af928d7ffd07330b28b95ae268469e6"} Nov 01 21:58:54.605477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.605420 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" event={"ID":"9feb9d45-8108-4a2f-af10-eaa3a10531ec","Type":"ContainerStarted","Data":"3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a"} Nov 01 21:58:54.611294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.609290 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" event={"ID":"73b67b27-c12c-49ae-aa4c-b043da0c8e15","Type":"ContainerStarted","Data":"bdb63bf046c568a094a77b72e339c99ba50034abc3f7335e201cba2de23993a0"} Nov 01 21:58:54.611294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.609339 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" event={"ID":"73b67b27-c12c-49ae-aa4c-b043da0c8e15","Type":"ContainerStarted","Data":"6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c"} Nov 01 21:58:54.614172 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.614144 5036 generic.go:356] "Generic (PLEG): container finished" podID="3385685f-48dd-41cf-b011-705b4e04689e" containerID="74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40" exitCode=137 Nov 01 21:58:54.614462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.614424 5036 scope.go:122] "RemoveContainer" containerID="74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40" Nov 01 21:58:54.616659 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.614766 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-f1e3c7" Nov 01 21:58:54.626279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.626252 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:54.633032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.630161 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" event={"ID":"1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e","Type":"ContainerDied","Data":"9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771"} Nov 01 21:58:54.633032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.630335 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc" Nov 01 21:58:54.633366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.633334 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9449574ed3ba22a40e17115b66fa8f31e3112b95a3677a8b595642d9cde0b771" Nov 01 21:58:54.676106 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.671935 5036 scope.go:122] "RemoveContainer" containerID="74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40" Nov 01 21:58:54.676532 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:54.676364 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40\": container with ID starting with 74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40 not found: ID does not exist" containerID="74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40" Nov 01 21:58:54.676532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.676407 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40"} err="failed to get container status \"74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40\": rpc error: code = NotFound desc = could not find container \"74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40\": container with ID starting with 74838f13a746a0fc4dfdd77d35cb80d154b40299ae22b562550031bbf23bcf40 not found: ID does not exist" Nov 01 21:58:54.685856 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.685792 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" podStartSLOduration=1.6857729780000001 podStartE2EDuration="1.685772978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:54.629737529 +0000 UTC m=+888.322718018" watchObservedRunningTime="2026-11-01 21:58:54.685772978 +0000 UTC m=+888.378753465" Nov 01 21:58:54.721939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.721824 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" podStartSLOduration=1.721799292 podStartE2EDuration="1.721799292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:54.658834012 +0000 UTC m=+888.351814499" watchObservedRunningTime="2026-11-01 21:58:54.721799292 +0000 UTC m=+888.414779836" Nov 01 21:58:54.795611 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:54.792037 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:55.403918 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:55.403815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:55.734469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:55.733164 5036 generic.go:356] "Generic (PLEG): container finished" podID="9feb9d45-8108-4a2f-af10-eaa3a10531ec" containerID="a472bb2e7ade8eebacb2510486f7a0ae3af928d7ffd07330b28b95ae268469e6" exitCode=0 Nov 01 21:58:55.734469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:55.733388 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" event={"ID":"9feb9d45-8108-4a2f-af10-eaa3a10531ec","Type":"ContainerDied","Data":"a472bb2e7ade8eebacb2510486f7a0ae3af928d7ffd07330b28b95ae268469e6"} Nov 01 21:58:55.737483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:55.736516 5036 generic.go:356] "Generic (PLEG): container finished" podID="73b67b27-c12c-49ae-aa4c-b043da0c8e15" containerID="bdb63bf046c568a094a77b72e339c99ba50034abc3f7335e201cba2de23993a0" exitCode=0 Nov 01 21:58:55.737483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:55.736802 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" event={"ID":"73b67b27-c12c-49ae-aa4c-b043da0c8e15","Type":"ContainerDied","Data":"bdb63bf046c568a094a77b72e339c99ba50034abc3f7335e201cba2de23993a0"} Nov 01 21:58:56.187483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.186528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.326741 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.325723 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:56.365620 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.364630 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:56.416145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.415563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:56.439523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.435607 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.876651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:56.876595 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.101388 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.101351 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:57.124058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.124021 5036 generic.go:356] "Generic (PLEG): container finished" podID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" containerID="8e65461a2f253219916aa00fb25634e69259a31dd4e4fd71e5f94b569e115280" exitCode=137 Nov 01 21:58:57.147793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.147749 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.148860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.148826 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.193536 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.193091 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.217153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.202629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 21:58:57.236469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.218465 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:58:57.355469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.339544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.355469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.339981 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:58:57.355469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.340696 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:57.382907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.372602 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:57.390596 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.390563 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:58:57.417022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.416961 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.424040 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.424011 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:58:57.448573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.440642 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:57.448573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:57.441216 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:58:58.180350 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.145618 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:58:58.181195 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.181155 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.187473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.184178 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:58:58.212469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.187759 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:58:58.216212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.199425 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.240644 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.209883 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:58.240897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.240868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.241052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.215166 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:58.268469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.243179 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:58:58.295798 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.215464 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.361689 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.361640 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.386310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.386252 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:58:58.391680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.391650 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:58:58.449757 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.213030 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.451393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.451367 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:58:58.516477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.516420 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:58:58.711986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.634016 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:58.712323 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.712282 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.712553 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.689980 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:58.728748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.728715 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.856123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.856083 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.892893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:58.860377 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\" (UniqueName: \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\") pod \"bbd8a474-9e5c-4aec-936a-b8ca8df24512\" (UID: \"bbd8a474-9e5c-4aec-936a-b8ca8df24512\") " Nov 01 21:58:59.154058 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.154019 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.218911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.218573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.222275 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.222230 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:58:59.247799 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.247611 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:58:59.247966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.247919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:58:59.254466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.251144 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc" pod "bbd8a474-9e5c-4aec-936a-b8ca8df24512" (UID: "bbd8a474-9e5c-4aec-936a-b8ca8df24512"). InnerVolumeSpecName "kube-api-access-w5qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:59.376662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.373379 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.385539 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:59.383081 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6944-8f4772\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:59.390175 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.390127 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.480521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.478479 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5qbc\" (UniqueName: \"kubernetes.io/projected/bbd8a474-9e5c-4aec-936a-b8ca8df24512-kube-api-access-w5qbc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:59.482351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.482309 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.497421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.497385 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" path="/var/lib/kubelet/pods/bbd8a474-9e5c-4aec-936a-b8ca8df24512/volumes" Nov 01 21:58:59.720069 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:59.720037 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.083s" Nov 01 21:58:59.721221 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.721198 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:58:59.721417 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.721403 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:58:59.746599 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.746535 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:58:59.746815 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.746800 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:58:59.746960 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.746945 5036 scope.go:122] "RemoveContainer" containerID="8e65461a2f253219916aa00fb25634e69259a31dd4e4fd71e5f94b569e115280" Nov 01 21:58:59.764464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.518812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:59.770574 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.738491 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.834579 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.736531 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:59.836581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.544557 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:58:59.836581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.834810 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.836581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.834653 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:58:59.836581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.693559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.869277 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.869252 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:58:59.869923 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.869899 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:58:59.872259 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.872242 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:58:59.885527 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:58:59.885494 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:58:59.920195 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:58:59.919342 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6944-8f4772\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:59:00.045938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.045907 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:59:00.048021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.047994 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:59:00.067692 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:00.067656 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6944-8f4772\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:59:00.070529 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:00.070498 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6944-8f4772\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:59:00.193650 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193614 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\") pod \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " Nov 01 21:59:00.193850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\") pod \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " Nov 01 21:59:00.193850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\") pod \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " Nov 01 21:59:00.193850 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193806 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\" (UniqueName: \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\") pod \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " Nov 01 21:59:00.194011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\") pod \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\" (UID: \"73b67b27-c12c-49ae-aa4c-b043da0c8e15\") " Nov 01 21:59:00.194011 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.193903 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\" (UniqueName: \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\") pod \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\" (UID: \"9feb9d45-8108-4a2f-af10-eaa3a10531ec\") " Nov 01 21:59:00.194111 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.194027 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca" pod "9feb9d45-8108-4a2f-af10-eaa3a10531ec" (UID: "9feb9d45-8108-4a2f-af10-eaa3a10531ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:00.194360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.194334 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca" pod "73b67b27-c12c-49ae-aa4c-b043da0c8e15" (UID: "73b67b27-c12c-49ae-aa4c-b043da0c8e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:00.194759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.194732 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.194759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.194760 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.196695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.196667 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca" pod "9feb9d45-8108-4a2f-af10-eaa3a10531ec" (UID: "9feb9d45-8108-4a2f-af10-eaa3a10531ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:00.196964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.196726 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc" pod "73b67b27-c12c-49ae-aa4c-b043da0c8e15" (UID: "73b67b27-c12c-49ae-aa4c-b043da0c8e15"). InnerVolumeSpecName "kube-api-access-s6kcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:00.196964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.196765 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d" pod "9feb9d45-8108-4a2f-af10-eaa3a10531ec" (UID: "9feb9d45-8108-4a2f-af10-eaa3a10531ec"). InnerVolumeSpecName "kube-api-access-8qn6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:00.196964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.196926 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca" pod "73b67b27-c12c-49ae-aa4c-b043da0c8e15" (UID: "73b67b27-c12c-49ae-aa4c-b043da0c8e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:00.231075 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231042 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv"] Nov 01 21:59:00.231700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231684 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" containerName="revoke-keys" Nov 01 21:59:00.231700 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231700 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" containerName="revoke-keys" Nov 01 21:59:00.231820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231729 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9feb9d45-8108-4a2f-af10-eaa3a10531ec" containerName="revoke-keys" Nov 01 21:59:00.231820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231735 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9feb9d45-8108-4a2f-af10-eaa3a10531ec" containerName="revoke-keys" Nov 01 21:59:00.231820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231767 5036 cpu_manager.go:414] "containerMap: removing container" podUID="73b67b27-c12c-49ae-aa4c-b043da0c8e15" containerName="revoke-keys" Nov 01 21:59:00.231820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231773 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b67b27-c12c-49ae-aa4c-b043da0c8e15" containerName="revoke-keys" Nov 01 21:59:00.231996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231887 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b67b27-c12c-49ae-aa4c-b043da0c8e15" containerName="revoke-keys" Nov 01 21:59:00.231996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231895 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" containerName="revoke-keys" Nov 01 21:59:00.231996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.231904 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9feb9d45-8108-4a2f-af10-eaa3a10531ec" containerName="revoke-keys" Nov 01 21:59:00.236267 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.236244 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.239095 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:00.239053 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6944-8f4772\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="bbd8a474-9e5c-4aec-936a-b8ca8df24512" pod="opendatahub/test-curl-6944-8f4772" Nov 01 21:59:00.246675 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.246656 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv"] Nov 01 21:59:00.296090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.296063 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b67b27-c12c-49ae-aa4c-b043da0c8e15-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.296090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.296090 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9feb9d45-8108-4a2f-af10-eaa3a10531ec-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.296283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.296103 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kcc\" (UniqueName: \"kubernetes.io/projected/73b67b27-c12c-49ae-aa4c-b043da0c8e15-kube-api-access-s6kcc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.296283 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.296113 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qn6d\" (UniqueName: \"kubernetes.io/projected/9feb9d45-8108-4a2f-af10-eaa3a10531ec-kube-api-access-8qn6d\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:00.398105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.398071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.398292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.398133 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.398292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.398284 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nql\" (UniqueName: \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.485758 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.485727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:00.507858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.503186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.507858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.503291 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.507858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.503398 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nql\" (UniqueName: \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.513478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.512515 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.517607 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.517569 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nql\" (UniqueName: \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.528421 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.527903 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0a9228-wblgv\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.552999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.552570 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" Nov 01 21:59:00.552999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.552601 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4" event={"ID":"9feb9d45-8108-4a2f-af10-eaa3a10531ec","Type":"ContainerDied","Data":"3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a"} Nov 01 21:59:00.552999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.552667 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6fe41440fa0125e8f11678a4b3fb1939fdb972a947ee1c2549d768d6104a1a" Nov 01 21:59:00.556586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.556561 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:00.556852 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.556823 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" event={"ID":"73b67b27-c12c-49ae-aa4c-b043da0c8e15","Type":"ContainerDied","Data":"6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c"} Nov 01 21:59:00.556942 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.556889 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j" Nov 01 21:59:00.556942 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.556913 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3699d5283b6786c0caea6421826bd5408f59c44e8d893fefb780e7352f6b9c" Nov 01 21:59:00.860143 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:00.860107 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35d1417_27c7_499b_8212_107af3192ec1.slice/crio-a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a WatchSource:0}: Error finding container a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a: Status 404 returned error can't find the container with id a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a Nov 01 21:59:00.862877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.862856 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv"] Nov 01 21:59:00.900087 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:00.900055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:01.181349 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.181301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:01.247683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.247560 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:01.256927 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.256905 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.272811 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.271905 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:01.272811 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.272141 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-681db477-openshift-default-dockercfg-kmjdt\"" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.324991 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325169 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325198 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325254 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325300 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xndq\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325433 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325495 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.325545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.325538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.331339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.331305 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:01.426412 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426379 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426610 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426593 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426695 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426661 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426750 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xndq\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426823 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426864 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426938 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.426995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.426867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.427104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.427007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.427280 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.427258 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.427362 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.427287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.427732 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.427713 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.429051 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.429027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.429908 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.429886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.439319 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.439262 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.440227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.440183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xndq\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\") pod \"e2e-ait-681db477-openshift-default-74959dc87f-fddv9\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.477085 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.477051 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:01.570589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.570431 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" event={"ID":"a35d1417-27c7-499b-8212-107af3192ec1","Type":"ContainerStarted","Data":"d4732cf96f1b73e494bf71dd612af9a1c5f109ddaad6b8c34a6f716e91ca9901"} Nov 01 21:59:01.570589 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.570546 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" event={"ID":"a35d1417-27c7-499b-8212-107af3192ec1","Type":"ContainerStarted","Data":"a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a"} Nov 01 21:59:01.590414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.590380 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:01.601736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.601669 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" podStartSLOduration=1.601651882 podStartE2EDuration="1.601651882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:01.600171272 +0000 UTC m=+895.293151751" watchObservedRunningTime="2026-11-01 21:59:01.601651882 +0000 UTC m=+895.294632369" Nov 01 21:59:01.622643 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.622439 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:01.808686 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:01.800736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:02.063370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.063290 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:02.102568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.102359 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:02.102568 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:02.102335 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d936208_b896_4a9c_a225_7551d3902f49.slice/crio-5951bcb4a4f55c362dc9b74b166293236d781c491ffa912e1e377d572b8ff84e WatchSource:0}: Error finding container 5951bcb4a4f55c362dc9b74b166293236d781c491ffa912e1e377d572b8ff84e: Status 404 returned error can't find the container with id 5951bcb4a4f55c362dc9b74b166293236d781c491ffa912e1e377d572b8ff84e Nov 01 21:59:02.105379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.105340 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:02.105494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.105428 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:02.105556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.105489 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:02.189913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.189878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:02.382104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.382010 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:02.586468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.586290 5036 generic.go:356] "Generic (PLEG): container finished" podID="a35d1417-27c7-499b-8212-107af3192ec1" containerID="d4732cf96f1b73e494bf71dd612af9a1c5f109ddaad6b8c34a6f716e91ca9901" exitCode=0 Nov 01 21:59:02.586669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.586534 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" event={"ID":"a35d1417-27c7-499b-8212-107af3192ec1","Type":"ContainerDied","Data":"d4732cf96f1b73e494bf71dd612af9a1c5f109ddaad6b8c34a6f716e91ca9901"} Nov 01 21:59:02.589816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.589410 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" event={"ID":"9d936208-b896-4a9c-a225-7551d3902f49","Type":"ContainerStarted","Data":"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc"} Nov 01 21:59:02.589816 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.589486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" event={"ID":"9d936208-b896-4a9c-a225-7551d3902f49","Type":"ContainerStarted","Data":"5951bcb4a4f55c362dc9b74b166293236d781c491ffa912e1e377d572b8ff84e"} Nov 01 21:59:02.590494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.590424 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:02.633525 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.631586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:02.662490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.662410 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" podStartSLOduration=1.6623915249999999 podStartE2EDuration="1.662391525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:02.656096499 +0000 UTC m=+896.349076985" watchObservedRunningTime="2026-11-01 21:59:02.662391525 +0000 UTC m=+896.355372012" Nov 01 21:59:02.817638 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:02.817543 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:03.070387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.065295 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:03.597590 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.597558 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:03.603550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.602685 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:03.603550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.602770 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:03.626591 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.626554 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9"] Nov 01 21:59:03.634242 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.633782 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.646158 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.646125 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9"] Nov 01 21:59:03.650684 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.650609 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.650910 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.650889 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tdg\" (UniqueName: \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.651076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.651041 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.753148 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.753106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.753282 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.753214 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tdg\" (UniqueName: \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.753282 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.753268 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.754375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.754348 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.756788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.756760 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.767494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.767462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tdg\" (UniqueName: \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\") pod \"maas-api-revoke-sub-4fe5d4f5-5f7n9\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:03.836220 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.836191 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:03.854338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.854220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\") pod \"a35d1417-27c7-499b-8212-107af3192ec1\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " Nov 01 21:59:03.854338 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.854305 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\") pod \"a35d1417-27c7-499b-8212-107af3192ec1\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " Nov 01 21:59:03.854576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.854555 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\" (UniqueName: \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\") pod \"a35d1417-27c7-499b-8212-107af3192ec1\" (UID: \"a35d1417-27c7-499b-8212-107af3192ec1\") " Nov 01 21:59:03.856490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.854908 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca" pod "a35d1417-27c7-499b-8212-107af3192ec1" (UID: "a35d1417-27c7-499b-8212-107af3192ec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:03.856490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.855342 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:03.856933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.856907 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca" pod "a35d1417-27c7-499b-8212-107af3192ec1" (UID: "a35d1417-27c7-499b-8212-107af3192ec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:03.857298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.857275 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql" pod "a35d1417-27c7-499b-8212-107af3192ec1" (UID: "a35d1417-27c7-499b-8212-107af3192ec1"). InnerVolumeSpecName "kube-api-access-j7nql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:03.965033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.964988 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a35d1417-27c7-499b-8212-107af3192ec1-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:03.965033 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.965034 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nql\" (UniqueName: \"kubernetes.io/projected/a35d1417-27c7-499b-8212-107af3192ec1-kube-api-access-j7nql\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:03.967201 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:03.967172 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:04.067976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.067940 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:04.230528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.220999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:04.349387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.347121 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9"] Nov 01 21:59:04.613410 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.613288 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" event={"ID":"cd67b024-7667-4c62-9c17-aeb82144c8b7","Type":"ContainerStarted","Data":"5dae2a01d0363e7976e6b1b05008759837ade6eaa0eb8f031a7d9c98056dea9a"} Nov 01 21:59:04.613933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.613658 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:04.615329 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.615309 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" Nov 01 21:59:04.615496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.615325 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv" event={"ID":"a35d1417-27c7-499b-8212-107af3192ec1","Type":"ContainerDied","Data":"a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a"} Nov 01 21:59:04.615496 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.615370 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a85bd646ad5494c9d02ed6e327d2fc403f6877593f2e1638f4e1715fd9dded5a" Nov 01 21:59:04.750783 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:04.750743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:05.142762 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.142728 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:05.278168 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.278129 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:05.339613 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.339573 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:05.340343 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.340327 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a35d1417-27c7-499b-8212-107af3192ec1" containerName="revoke-keys" Nov 01 21:59:05.340386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.340345 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35d1417-27c7-499b-8212-107af3192ec1" containerName="revoke-keys" Nov 01 21:59:05.340502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.340490 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35d1417-27c7-499b-8212-107af3192ec1" containerName="revoke-keys" Nov 01 21:59:05.344500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.344483 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.350900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.349600 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-13364262\"" Nov 01 21:59:05.354980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.354951 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:05.421494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.414430 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.421494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.414646 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zpc\" (UniqueName: \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.421494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.419744 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:05.544470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.522429 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.544470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.522769 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zpc\" (UniqueName: \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.544470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.534773 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.628468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.628189 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zpc\" (UniqueName: \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\") pod \"payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.677544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.672826 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:05.747379 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.746886 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd67b024-7667-4c62-9c17-aeb82144c8b7" containerID="485a7e4e33adbb9f82ddaf1bfe0992769e8276088eaad01b3e008c28d2694065" exitCode=0 Nov 01 21:59:05.748564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.748512 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" event={"ID":"cd67b024-7667-4c62-9c17-aeb82144c8b7","Type":"ContainerDied","Data":"485a7e4e33adbb9f82ddaf1bfe0992769e8276088eaad01b3e008c28d2694065"} Nov 01 21:59:05.809054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.808994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:05.940807 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:05.940426 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:06.054035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.054000 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:06.092736 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.092705 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:06.492801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.492770 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:06.531881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.531848 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:06.537251 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.537222 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.544237 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.544211 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-681db477\"" Nov 01 21:59:06.544585 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.544566 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-681db477-dockercfg-gblmm\"" Nov 01 21:59:06.575563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.575524 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:06.634830 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.634794 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:06.657656 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.657616 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.657841 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.657746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5p6l\" (UniqueName: \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.658586 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.658563 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:59:06.765506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.761395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.765506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.761555 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5p6l\" (UniqueName: \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.765506 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.762981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.831538 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.831409 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 21:59:06.841855 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.835280 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" event={"ID":"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac","Type":"ContainerStarted","Data":"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338"} Nov 01 21:59:06.842072 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.842048 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" event={"ID":"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac","Type":"ContainerStarted","Data":"ed451909365eb9dea9a688d49a48318f105ac81016cc4120a0d6c45e4372d368"} Nov 01 21:59:06.842226 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.842211 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:06.864253 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.863396 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5p6l\" (UniqueName: \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\") pod \"payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:06.865939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:06.865910 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:07.317070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.316859 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:07.342842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.340511 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" podStartSLOduration=2.340489747 podStartE2EDuration="2.340489747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:06.865765842 +0000 UTC m=+900.558746330" watchObservedRunningTime="2026-11-01 21:59:07.340489747 +0000 UTC m=+901.033470238" Nov 01 21:59:07.400523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.400490 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:07.504984 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.504948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\") pod \"cd67b024-7667-4c62-9c17-aeb82144c8b7\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " Nov 01 21:59:07.505220 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.505110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\") pod \"cd67b024-7667-4c62-9c17-aeb82144c8b7\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " Nov 01 21:59:07.505307 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.505247 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\" (UniqueName: \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\") pod \"cd67b024-7667-4c62-9c17-aeb82144c8b7\" (UID: \"cd67b024-7667-4c62-9c17-aeb82144c8b7\") " Nov 01 21:59:07.505457 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.505393 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca" pod "cd67b024-7667-4c62-9c17-aeb82144c8b7" (UID: "cd67b024-7667-4c62-9c17-aeb82144c8b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:07.505901 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.505874 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:07.508211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.508185 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg" pod "cd67b024-7667-4c62-9c17-aeb82144c8b7" (UID: "cd67b024-7667-4c62-9c17-aeb82144c8b7"). InnerVolumeSpecName "kube-api-access-b7tdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:07.508310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.508273 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca" pod "cd67b024-7667-4c62-9c17-aeb82144c8b7" (UID: "cd67b024-7667-4c62-9c17-aeb82144c8b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:07.539939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.539914 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7"] Nov 01 21:59:07.540863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.540839 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd67b024-7667-4c62-9c17-aeb82144c8b7" containerName="revoke-keys" Nov 01 21:59:07.540863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.540863 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd67b024-7667-4c62-9c17-aeb82144c8b7" containerName="revoke-keys" Nov 01 21:59:07.541090 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.541076 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd67b024-7667-4c62-9c17-aeb82144c8b7" containerName="revoke-keys" Nov 01 21:59:07.563022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.560722 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.577764 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.577695 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7"] Nov 01 21:59:07.607877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.607847 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gczn\" (UniqueName: \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.608248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.608224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.608561 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.608525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.608828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.608808 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd67b024-7667-4c62-9c17-aeb82144c8b7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:07.608988 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.608972 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tdg\" (UniqueName: \"kubernetes.io/projected/cd67b024-7667-4c62-9c17-aeb82144c8b7-kube-api-access-b7tdg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:07.637691 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.637661 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:07.655817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.655779 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:07.660892 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.660866 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.671323 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.671301 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-93a347eb\"" Nov 01 21:59:07.671665 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.671645 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:07.682335 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.682311 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:07.695592 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.695565 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.702317 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.702292 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-93a347eb-dockercfg-6rlcx\"" Nov 01 21:59:07.703581 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.703498 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:07.710234 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2m2\" (UniqueName: \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.710367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710326 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gczn\" (UniqueName: \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.710616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.710616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710407 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4zn\" (UniqueName: \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.710616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.710616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.710842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.710634 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.711486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.711463 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.725142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.725106 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gczn\" (UniqueName: \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.726737 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.726712 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972d2493-ptph7\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.770681 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.770648 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:07.811999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.811952 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.832542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.830522 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2m2\" (UniqueName: \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.832542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.830686 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.832542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.830720 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4zn\" (UniqueName: \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.832542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.830218 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.832542 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.832208 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.858279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.858245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2m2\" (UniqueName: \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\") pod \"payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:07.861066 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.861041 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4zn\" (UniqueName: \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\") pod \"payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:07.888744 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.888704 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:07.944303 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.944264 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" event={"ID":"cd67b024-7667-4c62-9c17-aeb82144c8b7","Type":"ContainerDied","Data":"5dae2a01d0363e7976e6b1b05008759837ade6eaa0eb8f031a7d9c98056dea9a"} Nov 01 21:59:07.944578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.944556 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dae2a01d0363e7976e6b1b05008759837ade6eaa0eb8f031a7d9c98056dea9a" Nov 01 21:59:07.944780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.944765 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9" Nov 01 21:59:07.969652 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.969617 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" event={"ID":"068c06a1-d6ac-4c39-95e6-792c876ffa6d","Type":"ContainerStarted","Data":"7f48e919e1fdc7f625c0da883daf54245c2ab81e81e474b0347e086a224b59bd"} Nov 01 21:59:07.975526 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.975498 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" event={"ID":"068c06a1-d6ac-4c39-95e6-792c876ffa6d","Type":"ContainerStarted","Data":"47b965b85c976b9c2097ea7d044a919213bccf2f103f747ad5ecf3da48417701"} Nov 01 21:59:07.975765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.975750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:07.982223 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:07.982182 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:08.039671 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.039638 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:08.053310 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.052622 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" podStartSLOduration=2.052602384 podStartE2EDuration="2.052602384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:07.993890104 +0000 UTC m=+901.686870592" watchObservedRunningTime="2026-11-01 21:59:08.052602384 +0000 UTC m=+901.745582871" Nov 01 21:59:08.405860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.405811 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:08.635834 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:08.635800 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42ea635_8050_459b_bf1c_ba629317bfb0.slice/crio-9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81 WatchSource:0}: Error finding container 9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81: Status 404 returned error can't find the container with id 9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81 Nov 01 21:59:08.638801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.638773 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7"] Nov 01 21:59:08.657240 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:08.657208 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17afae5b_e30e_4d5f_93ea_4b03010a5cfa.slice/crio-a4641a650b1f891e4a40ceaf5d058aca2ce0f26c527ed7b2892b87919b2889bb WatchSource:0}: Error finding container a4641a650b1f891e4a40ceaf5d058aca2ce0f26c527ed7b2892b87919b2889bb: Status 404 returned error can't find the container with id a4641a650b1f891e4a40ceaf5d058aca2ce0f26c527ed7b2892b87919b2889bb Nov 01 21:59:08.657669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.657646 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:08.659484 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.659421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:08.989222 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.989146 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" event={"ID":"ef46ac92-7da8-47ba-9a8b-091fb244558d","Type":"ContainerStarted","Data":"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347"} Nov 01 21:59:08.989478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.989453 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" event={"ID":"ef46ac92-7da8-47ba-9a8b-091fb244558d","Type":"ContainerStarted","Data":"eed9338a8df1a79369c13fce38263f269b54af81d9baca4526a1290d3cea0801"} Nov 01 21:59:08.990298 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.990278 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:08.993235 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.993196 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" event={"ID":"c42ea635-8050-459b-bf1c-ba629317bfb0","Type":"ContainerStarted","Data":"9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81"} Nov 01 21:59:09.000006 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:08.999974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" event={"ID":"17afae5b-e30e-4d5f-93ea-4b03010a5cfa","Type":"ContainerStarted","Data":"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a"} Nov 01 21:59:09.000124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:09.000034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" event={"ID":"17afae5b-e30e-4d5f-93ea-4b03010a5cfa","Type":"ContainerStarted","Data":"a4641a650b1f891e4a40ceaf5d058aca2ce0f26c527ed7b2892b87919b2889bb"} Nov 01 21:59:09.000194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:09.000177 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:09.012722 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:09.012673 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" podStartSLOduration=2.012657493 podStartE2EDuration="2.012657493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:09.009237113 +0000 UTC m=+902.702217605" watchObservedRunningTime="2026-11-01 21:59:09.012657493 +0000 UTC m=+902.705637981" Nov 01 21:59:09.029399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:09.029355 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" podStartSLOduration=2.029339471 podStartE2EDuration="2.029339471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:09.027112998 +0000 UTC m=+902.720093486" watchObservedRunningTime="2026-11-01 21:59:09.029339471 +0000 UTC m=+902.722319958" Nov 01 21:59:09.538780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:09.538744 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:10.010435 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.010395 5036 generic.go:356] "Generic (PLEG): container finished" podID="c42ea635-8050-459b-bf1c-ba629317bfb0" containerID="716519b3de85e19f64cdeff6916de03801d49e046e29cfde0504aaa0607cdb9e" exitCode=0 Nov 01 21:59:10.010937 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.010508 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" event={"ID":"c42ea635-8050-459b-bf1c-ba629317bfb0","Type":"ContainerDied","Data":"716519b3de85e19f64cdeff6916de03801d49e046e29cfde0504aaa0607cdb9e"} Nov 01 21:59:10.384548 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.384507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:10.466544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.466510 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk"] Nov 01 21:59:10.471716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.471678 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.505683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.495255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.505683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.495328 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2sc\" (UniqueName: \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.505683 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.495473 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.509494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.506513 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk"] Nov 01 21:59:10.596725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.596684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.596951 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.596933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.596999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.596987 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2sc\" (UniqueName: \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.597697 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.597669 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.599377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.599350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.609916 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.609889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2sc\" (UniqueName: \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\") pod \"maas-api-revoke-sub-135be121-cjlmk\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:10.810403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:10.810370 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:11.109360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.109328 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk"] Nov 01 21:59:11.122069 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:11.122034 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1481809f_ee9d_4668_8b43_09833840840c.slice/crio-a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791 WatchSource:0}: Error finding container a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791: Status 404 returned error can't find the container with id a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791 Nov 01 21:59:11.146142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.146111 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:11.183072 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.183048 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:11.216035 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.215998 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\") pod \"c42ea635-8050-459b-bf1c-ba629317bfb0\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " Nov 01 21:59:11.216192 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.216111 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\" (UniqueName: \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\") pod \"c42ea635-8050-459b-bf1c-ba629317bfb0\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " Nov 01 21:59:11.216192 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.216152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\") pod \"c42ea635-8050-459b-bf1c-ba629317bfb0\" (UID: \"c42ea635-8050-459b-bf1c-ba629317bfb0\") " Nov 01 21:59:11.223577 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.223531 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca" pod "c42ea635-8050-459b-bf1c-ba629317bfb0" (UID: "c42ea635-8050-459b-bf1c-ba629317bfb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:11.278005 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.277967 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn" pod "c42ea635-8050-459b-bf1c-ba629317bfb0" (UID: "c42ea635-8050-459b-bf1c-ba629317bfb0"). InnerVolumeSpecName "kube-api-access-8gczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:11.278307 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.278270 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca" pod "c42ea635-8050-459b-bf1c-ba629317bfb0" (UID: "c42ea635-8050-459b-bf1c-ba629317bfb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:11.319007 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.318974 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:11.319007 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.319002 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gczn\" (UniqueName: \"kubernetes.io/projected/c42ea635-8050-459b-bf1c-ba629317bfb0-kube-api-access-8gczn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:11.319007 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.319012 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c42ea635-8050-459b-bf1c-ba629317bfb0-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:11.833946 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:11.833914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:12.033170 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.033137 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" Nov 01 21:59:12.033170 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.033156 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7" event={"ID":"c42ea635-8050-459b-bf1c-ba629317bfb0","Type":"ContainerDied","Data":"9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81"} Nov 01 21:59:12.033399 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.033203 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7956fed1134c0788f7d3fe126170b31445bd296706273708e4a72fbd918e81" Nov 01 21:59:12.035318 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.035289 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" event={"ID":"1481809f-ee9d-4668-8b43-09833840840c","Type":"ContainerStarted","Data":"6ad7b2f8f6534b059704df9934137e8b5b08faf37fb48dbb88519b81b43b3bf5"} Nov 01 21:59:12.035512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.035332 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" event={"ID":"1481809f-ee9d-4668-8b43-09833840840c","Type":"ContainerStarted","Data":"a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791"} Nov 01 21:59:12.056425 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.056374 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" podStartSLOduration=2.056361797 podStartE2EDuration="2.056361797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:12.054042722 +0000 UTC m=+905.747023208" watchObservedRunningTime="2026-11-01 21:59:12.056361797 +0000 UTC m=+905.749342283" Nov 01 21:59:12.552480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.551011 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:12.678477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.678000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:12.813015 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:12.812922 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:13.046885 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:13.046852 5036 generic.go:356] "Generic (PLEG): container finished" podID="1481809f-ee9d-4668-8b43-09833840840c" containerID="6ad7b2f8f6534b059704df9934137e8b5b08faf37fb48dbb88519b81b43b3bf5" exitCode=0 Nov 01 21:59:13.047125 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:13.046921 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" event={"ID":"1481809f-ee9d-4668-8b43-09833840840c","Type":"ContainerDied","Data":"6ad7b2f8f6534b059704df9934137e8b5b08faf37fb48dbb88519b81b43b3bf5"} Nov 01 21:59:13.614240 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:13.614210 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:13.732083 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:13.732045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:13.950306 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:13.950212 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:14.233529 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.228870 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:14.544871 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.544838 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:14.585931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.579570 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\" (UniqueName: \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\") pod \"1481809f-ee9d-4668-8b43-09833840840c\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " Nov 01 21:59:14.585931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.579691 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\") pod \"1481809f-ee9d-4668-8b43-09833840840c\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " Nov 01 21:59:14.585931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.579816 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\") pod \"1481809f-ee9d-4668-8b43-09833840840c\" (UID: \"1481809f-ee9d-4668-8b43-09833840840c\") " Nov 01 21:59:14.585931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.580570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca" pod "1481809f-ee9d-4668-8b43-09833840840c" (UID: "1481809f-ee9d-4668-8b43-09833840840c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:14.585931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.584190 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc" pod "1481809f-ee9d-4668-8b43-09833840840c" (UID: "1481809f-ee9d-4668-8b43-09833840840c"). InnerVolumeSpecName "kube-api-access-sk2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:14.598873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.598838 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca" pod "1481809f-ee9d-4668-8b43-09833840840c" (UID: "1481809f-ee9d-4668-8b43-09833840840c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:14.682456 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.682405 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1481809f-ee9d-4668-8b43-09833840840c-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:14.682942 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.682486 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1481809f-ee9d-4668-8b43-09833840840c-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:14.682942 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.682508 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk2sc\" (UniqueName: \"kubernetes.io/projected/1481809f-ee9d-4668-8b43-09833840840c-kube-api-access-sk2sc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:14.753893 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:14.753754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:15.078725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.078683 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" event={"ID":"1481809f-ee9d-4668-8b43-09833840840c","Type":"ContainerDied","Data":"a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791"} Nov 01 21:59:15.078900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.078782 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4aaaee7d025a3a07c3541d0350c004ecb1e93a8fccea1b30808143871618791" Nov 01 21:59:15.078900 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.078864 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk" Nov 01 21:59:15.360649 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.360556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:15.490313 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.490275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:15.635511 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.632185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:15.768926 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.768874 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:15.898076 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:15.897975 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:16.481788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:16.481754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:16.641007 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:16.640973 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:17.018992 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.018950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:17.816232 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.816191 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:17.857314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.857281 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd"] Nov 01 21:59:17.858632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858604 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1481809f-ee9d-4668-8b43-09833840840c" containerName="revoke-keys" Nov 01 21:59:17.858632 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858631 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1481809f-ee9d-4668-8b43-09833840840c" containerName="revoke-keys" Nov 01 21:59:17.858873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858678 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c42ea635-8050-459b-bf1c-ba629317bfb0" containerName="revoke-keys" Nov 01 21:59:17.858873 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858686 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42ea635-8050-459b-bf1c-ba629317bfb0" containerName="revoke-keys" Nov 01 21:59:17.858976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858903 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1481809f-ee9d-4668-8b43-09833840840c" containerName="revoke-keys" Nov 01 21:59:17.858976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.858920 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42ea635-8050-459b-bf1c-ba629317bfb0" containerName="revoke-keys" Nov 01 21:59:17.864287 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.864262 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:17.869048 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.869021 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:59:17.869163 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.869101 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:59:17.877067 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.877035 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd"] Nov 01 21:59:17.910483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.903735 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:59:17.910483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.903864 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:59:17.956780 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.953673 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsvq\" (UniqueName: \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:17.967576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.966645 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:17.967576 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.966726 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:17.976124 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.975890 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:17.990710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.990677 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 21:59:17.998290 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:17.998263 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.006213 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.006181 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 21:59:18.029732 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.024812 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.035370 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.035302 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 21:59:18.053137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.052886 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 21:59:18.070361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.070293 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.070546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.070488 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsvq\" (UniqueName: \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.070546 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.070533 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.070713 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.070690 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.070931 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.070912 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.071068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071051 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.071200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.071314 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071301 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b92r\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.071476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59xs\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.071662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071645 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.071788 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071773 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.071909 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.071896 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.072023 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072009 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.072126 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.072256 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072236 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.072366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.072521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072505 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.072678 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072628 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.072842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072826 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.072945 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.072932 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.073050 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.073037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.074759 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.074734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.080466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.078198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.090381 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.090354 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsvq\" (UniqueName: \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\") pod \"maas-api-revoke-sub-956642ac-f5kwd\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.174738 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174704 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.174899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174748 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.174899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.174899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b92r\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.174899 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174879 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59xs\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174931 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.174971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175051 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175135 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175129 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175170 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175267 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175298 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175323 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175386 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175524 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175749 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.175938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175846 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.175938 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.175919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.176225 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.176038 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.179547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.176579 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.179547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.176856 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.179547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.177176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.179547 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.177265 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.181544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.181490 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.185008 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.184963 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.190194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.190169 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.194717 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.194692 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.196881 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.196859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.197205 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.197183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.197468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.197422 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b92r\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.199781 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.199758 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59xs\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.216751 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.216728 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:18.340218 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.340142 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd"] Nov 01 21:59:18.347953 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.347928 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.350868 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.350830 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd"] Nov 01 21:59:18.370545 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.370518 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:18.379149 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.379125 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.379334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.379317 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.379485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.379342 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:18.379749 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.379732 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7lxs\" (UniqueName: \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.481396 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.481361 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.483214 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.483192 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.484019 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.483999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7lxs\" (UniqueName: \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.485387 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.482935 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.489022 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.488978 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.498268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.498220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7lxs\" (UniqueName: \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\") pod \"maas-api-revoke-sub-0946ec3d-kgfrd\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.530595 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:18.530567 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e75bfbb_6c3a_4132_9773_24a6257f5795.slice/crio-fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47 WatchSource:0}: Error finding container fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47: Status 404 returned error can't find the container with id fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47 Nov 01 21:59:18.533765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.533733 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd"] Nov 01 21:59:18.673246 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.673183 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:18.692680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.692657 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 21:59:18.693139 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:18.693109 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fea9e49_362e_4da9_a509_4f051892f91c.slice/crio-3ff7ff91a64d54813eac958cad2407fa8485fbca0a71ebea5ffc1ce17626793b WatchSource:0}: Error finding container 3ff7ff91a64d54813eac958cad2407fa8485fbca0a71ebea5ffc1ce17626793b: Status 404 returned error can't find the container with id 3ff7ff91a64d54813eac958cad2407fa8485fbca0a71ebea5ffc1ce17626793b Nov 01 21:59:18.694171 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:18.694103 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6ca82_79fd_4d85_a62e_d6e3310e7852.slice/crio-0e0b86e7dacb8f6a80b6c6c2007e16ea46d445f9749b172b06a49ef0865c6151 WatchSource:0}: Error finding container 0e0b86e7dacb8f6a80b6c6c2007e16ea46d445f9749b172b06a49ef0865c6151: Status 404 returned error can't find the container with id 0e0b86e7dacb8f6a80b6c6c2007e16ea46d445f9749b172b06a49ef0865c6151 Nov 01 21:59:18.697877 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.697842 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.697979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.697906 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.697979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.697938 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.698082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.697980 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.698082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.697984 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.698082 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.698032 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:18.700860 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.700827 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 21:59:18.717924 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:18.717851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:18.968868 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:18.968834 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:59:19.019947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.019905 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd"] Nov 01 21:59:19.148230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.147387 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" event={"ID":"3fea9e49-362e-4da9-a509-4f051892f91c","Type":"ContainerStarted","Data":"d3342f0d5d5a08dfc838b0a96c4692acae635f9b1c94c4dd738591eb6ec0f8e0"} Nov 01 21:59:19.148230 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.147466 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" event={"ID":"3fea9e49-362e-4da9-a509-4f051892f91c","Type":"ContainerStarted","Data":"3ff7ff91a64d54813eac958cad2407fa8485fbca0a71ebea5ffc1ce17626793b"} Nov 01 21:59:19.149814 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.149421 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" event={"ID":"558aea62-4f7c-47bd-b406-3c8a59a2769b","Type":"ContainerStarted","Data":"f89236f3f0ef9f526124d4bfeb5cd720a8f10b5ae7c9123614a93ec16107d860"} Nov 01 21:59:19.151978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.151658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" event={"ID":"2e75bfbb-6c3a-4132-9773-24a6257f5795","Type":"ContainerStarted","Data":"29bcd317f8f604c4fbb8388ceb3915f5b36afdfea7cffdf18d35b815bed43e83"} Nov 01 21:59:19.151978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.151706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" event={"ID":"2e75bfbb-6c3a-4132-9773-24a6257f5795","Type":"ContainerStarted","Data":"fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47"} Nov 01 21:59:19.153896 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.153869 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" event={"ID":"23c6ca82-79fd-4d85-a62e-d6e3310e7852","Type":"ContainerStarted","Data":"7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231"} Nov 01 21:59:19.153979 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.153920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" event={"ID":"23c6ca82-79fd-4d85-a62e-d6e3310e7852","Type":"ContainerStarted","Data":"0e0b86e7dacb8f6a80b6c6c2007e16ea46d445f9749b172b06a49ef0865c6151"} Nov 01 21:59:19.259502 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.256162 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podStartSLOduration=2.2561433 podStartE2EDuration="2.2561433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.253287822 +0000 UTC m=+912.946268319" watchObservedRunningTime="2026-11-01 21:59:19.2561433 +0000 UTC m=+912.949123788" Nov 01 21:59:19.303292 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.303228 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podStartSLOduration=2.303214466 podStartE2EDuration="2.303214466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.296947519 +0000 UTC m=+912.989928009" watchObservedRunningTime="2026-11-01 21:59:19.303214466 +0000 UTC m=+912.996195025" Nov 01 21:59:19.326302 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.326248 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" podStartSLOduration=2.326234656 podStartE2EDuration="2.326234656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.319799681 +0000 UTC m=+913.012780167" watchObservedRunningTime="2026-11-01 21:59:19.326234656 +0000 UTC m=+913.019215141" Nov 01 21:59:19.373296 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.373266 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:19.375385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.375358 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:19.375549 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.375420 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:19.380060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.380032 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:19.381155 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.381132 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:19.381262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.381183 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:19.536685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:19.536644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:20.043825 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.043796 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:20.044462 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.044427 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:20.272680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.259503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" event={"ID":"558aea62-4f7c-47bd-b406-3c8a59a2769b","Type":"ContainerStarted","Data":"2e2c7c43c3409e789fd4a151f9eb7904396df962a1984b8aa244627a9308861b"} Nov 01 21:59:20.272680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.263701 5036 generic.go:356] "Generic (PLEG): container finished" podID="2e75bfbb-6c3a-4132-9773-24a6257f5795" containerID="29bcd317f8f604c4fbb8388ceb3915f5b36afdfea7cffdf18d35b815bed43e83" exitCode=0 Nov 01 21:59:20.272680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.264351 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" event={"ID":"2e75bfbb-6c3a-4132-9773-24a6257f5795","Type":"ContainerDied","Data":"29bcd317f8f604c4fbb8388ceb3915f5b36afdfea7cffdf18d35b815bed43e83"} Nov 01 21:59:20.363898 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.363311 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" podStartSLOduration=2.363292516 podStartE2EDuration="2.363292516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:20.335911266 +0000 UTC m=+914.028891753" watchObservedRunningTime="2026-11-01 21:59:20.363292516 +0000 UTC m=+914.056273002" Nov 01 21:59:20.375728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.373950 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:20.375728 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.374003 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:20.402468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.402125 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:20.402468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.402181 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:20.456486 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:20.456015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:21.041472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.040083 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:21.247469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.235790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:21.375093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.374989 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:21.375741 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.375709 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:21.381047 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.381021 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:21.381224 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.381204 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:21.395789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.382610 5036 generic.go:356] "Generic (PLEG): container finished" podID="558aea62-4f7c-47bd-b406-3c8a59a2769b" containerID="2e2c7c43c3409e789fd4a151f9eb7904396df962a1984b8aa244627a9308861b" exitCode=0 Nov 01 21:59:21.395789 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:21.383274 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" event={"ID":"558aea62-4f7c-47bd-b406-3c8a59a2769b","Type":"ContainerDied","Data":"2e2c7c43c3409e789fd4a151f9eb7904396df962a1984b8aa244627a9308861b"} Nov 01 21:59:22.055865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.055818 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:22.060028 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.059999 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:22.162469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.161368 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\") pod \"2e75bfbb-6c3a-4132-9773-24a6257f5795\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " Nov 01 21:59:22.162469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.161472 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\" (UniqueName: \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\") pod \"2e75bfbb-6c3a-4132-9773-24a6257f5795\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " Nov 01 21:59:22.162469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.161578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\") pod \"2e75bfbb-6c3a-4132-9773-24a6257f5795\" (UID: \"2e75bfbb-6c3a-4132-9773-24a6257f5795\") " Nov 01 21:59:22.187475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.173738 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca" pod "2e75bfbb-6c3a-4132-9773-24a6257f5795" (UID: "2e75bfbb-6c3a-4132-9773-24a6257f5795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:22.187475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.174091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca" pod "2e75bfbb-6c3a-4132-9773-24a6257f5795" (UID: "2e75bfbb-6c3a-4132-9773-24a6257f5795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:22.305466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.261694 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq" pod "2e75bfbb-6c3a-4132-9773-24a6257f5795" (UID: "2e75bfbb-6c3a-4132-9773-24a6257f5795"). InnerVolumeSpecName "kube-api-access-hwsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:22.325842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.306900 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:22.325842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.306941 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsvq\" (UniqueName: \"kubernetes.io/projected/2e75bfbb-6c3a-4132-9773-24a6257f5795-kube-api-access-hwsvq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:22.325842 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.306959 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e75bfbb-6c3a-4132-9773-24a6257f5795-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:22.373991 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.373955 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:22.374185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.374008 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:22.433564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.401428 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:22.433564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.401508 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:22.456720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.456695 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" Nov 01 21:59:22.457793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.457761 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd" event={"ID":"2e75bfbb-6c3a-4132-9773-24a6257f5795","Type":"ContainerDied","Data":"fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47"} Nov 01 21:59:22.458003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.457815 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb101ac16e71b0524eb32ae333ce75e7bd93a368c3824f8f2da0b5bde668be47" Nov 01 21:59:22.756477 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.755385 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:22.891859 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:22.889577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:23.066605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.066524 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:23.140469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.124679 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:23.222088 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.222015 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\") pod \"558aea62-4f7c-47bd-b406-3c8a59a2769b\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " Nov 01 21:59:23.222384 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.222362 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\" (UniqueName: \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\") pod \"558aea62-4f7c-47bd-b406-3c8a59a2769b\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " Nov 01 21:59:23.222507 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.222415 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\") pod \"558aea62-4f7c-47bd-b406-3c8a59a2769b\" (UID: \"558aea62-4f7c-47bd-b406-3c8a59a2769b\") " Nov 01 21:59:23.264817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.253018 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca" pod "558aea62-4f7c-47bd-b406-3c8a59a2769b" (UID: "558aea62-4f7c-47bd-b406-3c8a59a2769b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:23.264817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.259865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs" pod "558aea62-4f7c-47bd-b406-3c8a59a2769b" (UID: "558aea62-4f7c-47bd-b406-3c8a59a2769b"). InnerVolumeSpecName "kube-api-access-h7lxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:23.264817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.260219 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca" pod "558aea62-4f7c-47bd-b406-3c8a59a2769b" (UID: "558aea62-4f7c-47bd-b406-3c8a59a2769b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:23.362467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.332130 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:23.362467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.332170 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558aea62-4f7c-47bd-b406-3c8a59a2769b-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:23.362467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.332189 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7lxs\" (UniqueName: \"kubernetes.io/projected/558aea62-4f7c-47bd-b406-3c8a59a2769b-kube-api-access-h7lxs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:23.376122 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.376088 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:23.390699 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.390659 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:23.411655 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.391091 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:23.411897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.411870 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:23.636556 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.636519 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 21:59:23.638203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.638177 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2e75bfbb-6c3a-4132-9773-24a6257f5795" containerName="revoke-keys" Nov 01 21:59:23.638360 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.638345 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e75bfbb-6c3a-4132-9773-24a6257f5795" containerName="revoke-keys" Nov 01 21:59:23.638534 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.638520 5036 cpu_manager.go:414] "containerMap: removing container" podUID="558aea62-4f7c-47bd-b406-3c8a59a2769b" containerName="revoke-keys" Nov 01 21:59:23.638636 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.638624 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558aea62-4f7c-47bd-b406-3c8a59a2769b" containerName="revoke-keys" Nov 01 21:59:23.638907 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.638892 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="558aea62-4f7c-47bd-b406-3c8a59a2769b" containerName="revoke-keys" Nov 01 21:59:23.639017 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.639008 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e75bfbb-6c3a-4132-9773-24a6257f5795" containerName="revoke-keys" Nov 01 21:59:23.652532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.651008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:23.657466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.654958 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" Nov 01 21:59:23.748720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.748692 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 21:59:23.749101 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.749077 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd" event={"ID":"558aea62-4f7c-47bd-b406-3c8a59a2769b","Type":"ContainerDied","Data":"f89236f3f0ef9f526124d4bfeb5cd720a8f10b5ae7c9123614a93ec16107d860"} Nov 01 21:59:23.749248 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.749234 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89236f3f0ef9f526124d4bfeb5cd720a8f10b5ae7c9123614a93ec16107d860" Nov 01 21:59:23.749347 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.749013 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.769353 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.768629 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1aee4437-gw-openshift-default-dockercfg-g22gm\"" Nov 01 21:59:23.847465 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.839498 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885616 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885739 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885806 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885872 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwh7l\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.885921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.886075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.886121 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:23.887562 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.886152 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.987683 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.987779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwh7l\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.987829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988006 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988137 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988165 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.988298 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:23.989219 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.004468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.004334 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.005739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.005324 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.005739 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.005626 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.005932 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.005881 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.117254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.117181 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwh7l\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.147123 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.147031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.188493 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.184011 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.214485 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.206166 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\") pod \"e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.223978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.223943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:24.226720 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.226697 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:24.378947 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.378906 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:24.379171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.378987 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:24.384500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.379651 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:24.384500 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.379694 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:24.744431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.744381 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:24.821483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.806848 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:24.904243 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:24.899528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:25.235531 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:25.217573 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eab6863_80f1_49e0_99b1_ac5a6b601000.slice/crio-a8035599dda09768c2753c772193ca7f206f2ff5bb453a074b579171e4852980 WatchSource:0}: Error finding container a8035599dda09768c2753c772193ca7f206f2ff5bb453a074b579171e4852980: Status 404 returned error can't find the container with id a8035599dda09768c2753c772193ca7f206f2ff5bb453a074b579171e4852980 Nov 01 21:59:25.299501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.283837 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:25.299501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.283933 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:25.299501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.283994 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:25.299501 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.294200 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 21:59:25.426582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.417586 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:25.426582 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.417644 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:25.472828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.456785 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:25.472828 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.456841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:25.625362 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.625285 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:25.875716 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:25.875684 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" event={"ID":"2eab6863-80f1-49e0-99b1-ac5a6b601000","Type":"ContainerStarted","Data":"a8035599dda09768c2753c772193ca7f206f2ff5bb453a074b579171e4852980"} Nov 01 21:59:26.373958 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.373915 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 21:59:26.374147 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.373966 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 21:59:26.380712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.380679 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 01 21:59:26.380852 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.380729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 01 21:59:26.449520 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.449473 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:26.893361 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.893317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" event={"ID":"2eab6863-80f1-49e0-99b1-ac5a6b601000","Type":"ContainerStarted","Data":"eaffbe29ecc78aab10fcc8be79b01219cb9a2d0008d4f41210ba23af442f0dab"} Nov 01 21:59:26.924295 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:26.924232 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" podStartSLOduration=3.924214583 podStartE2EDuration="3.924214583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:26.919478536 +0000 UTC m=+920.612459025" watchObservedRunningTime="2026-11-01 21:59:26.924214583 +0000 UTC m=+920.617195070" Nov 01 21:59:27.230858 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.230771 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:27.362784 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.360323 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:27.552268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.551827 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:27.759464 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.758532 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 21:59:27.948537 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.947882 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:27.978467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:27.950680 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 21:59:28.225420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.225344 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:28.226239 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.226218 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:28.226943 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.226921 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 21:59:28.337757 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.337724 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:28.338563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.338535 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:28.339408 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.339381 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 21:59:28.732468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.722088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:28.858466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:28.857944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:29.002523 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:29.001128 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:30.172466 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.172235 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:30.324075 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.324035 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:30.373531 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.373492 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd"] Nov 01 21:59:30.383463 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.381477 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.389812 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.389787 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:59:30.390091 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.390072 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:59:30.391560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.391000 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:59:30.391560 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.391147 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:59:30.440706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.430504 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd"] Nov 01 21:59:30.608480 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.563280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.677869 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.677818 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.678431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.678406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxln\" (UniqueName: \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.686068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.633727 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.686068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.627079 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:30.686068 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.660844 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.816497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.816460 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.816887 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.816862 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.817142 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.817125 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxln\" (UniqueName: \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.841630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.822573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:59:30.841630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.839345 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:30.841630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.841618 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:30.881060 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.880467 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.908490 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:30.888768 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.081704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.081610 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z"] Nov 01 21:59:31.238478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.237255 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.238478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.196321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:31.238478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.196917 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:59:31.238478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.196953 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 21:59:31.238478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.195495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxln\" (UniqueName: \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\") pod \"maas-api-revoke-keys-e2e-ait-13364262-b2pbd\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:31.257497 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.243915 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 01 21:59:31.284194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.279561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.284194 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.282633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.330367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.318143 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.330367 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.318054 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z"] Nov 01 21:59:31.349538 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.346694 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:31.397268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.396916 5036 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:59:31.397268 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.396991 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.398036 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.398007 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.398272 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.398249 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbb8\" (UniqueName: \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.423743 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.423717 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.552488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.544093 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.552488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.544148 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbb8\" (UniqueName: \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.552488 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.544394 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.623507 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.619419 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.715817 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.707897 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.774742 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.768848 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbb8\" (UniqueName: \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\") pod \"maas-api-revoke-keys-e2e-ait-681db477-8nb7z\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.823414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.790299 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.823414 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.791502 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:59:31.826171 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.824959 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.836473 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.826565 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:59:31.889521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.885766 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:31.955768 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.939838 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.955768 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.940807 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:59:31.983630 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:31.976550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.055612 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.028964 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:59:32.055612 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.029016 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt"] Nov 01 21:59:32.328478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.320627 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 21:59:32.328478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.322411 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 21:59:32.328478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.328151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.328478 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.328382 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.564609 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.445812 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.735437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.718859 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9ct\" (UniqueName: \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:32.735437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.719079 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:32.735437 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.719311 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:32.747564 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.747521 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.789669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.783987 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:32.854524 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.816457 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:59:32.903925 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.889716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 21:59:32.971471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.942768 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:59:32.971471 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:32.942840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.058704 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.058646 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:33.059010 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.058984 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9ct\" (UniqueName: \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:33.059364 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.059340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:33.115735 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.115627 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.146470 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.118020 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 21:59:33.325415 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.325366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.341467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.328069 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.358854 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.358813 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:59:33.360078 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.360049 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 21:59:33.634469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.625626 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 21:59:33.676472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.654219 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:33.822865 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.719025 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.967121 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.967032 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:59:34.031212 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.907334 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 21:59:34.050568 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.042234 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:34.142996 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.142945 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.143712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:33.905870 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:34.210347 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.519s" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210627 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210665 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt"] Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210812 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210876 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210934 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.210994 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.211046 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.211101 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.241599 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.251928 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.243711 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9ct\" (UniqueName: \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\") pod \"maas-api-revoke-keys-e2e-ait-93a347eb-stdlt\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:34.275870 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.259728 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:34.303664 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.303626 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.353098 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.340972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.366020 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.365964 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.367107 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.366998 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 21:59:34.627476 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.626568 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.745647 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.744300 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.834294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.834193 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 21:59:34.895801 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.895477 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 21:59:34.903698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.903674 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 21:59:34.904003 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.903987 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 21:59:34.953475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.950646 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 21:59:34.996727 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:34.996709 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:35.008690 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.008110 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 21:59:35.016662 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.016577 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:35.018209 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.017330 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 21:59:35.070334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.062776 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 21:59:35.070334 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.063741 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 21:59:35.070784 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.070571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 21:59:35.080492 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.076284 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 21:59:35.139672 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.139642 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 21:59:35.150516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.149797 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 21:59:35.150516 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.149916 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.155325 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.155059 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 21:59:35.155325 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.155236 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8pxdd\"" Nov 01 21:59:35.155325 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.155267 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 21:59:35.155325 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.155298 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 21:59:35.159532 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.159509 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd"] Nov 01 21:59:35.236018 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.235990 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z"] Nov 01 21:59:35.258934 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.258900 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259065 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259012 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259113 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259091 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259154 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259121 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259291 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259275 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259357 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259327 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.259403 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.259356 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzhn\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360279 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360366 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360341 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360400 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360584 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.360929 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.360601 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzhn\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.361266 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.361236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.363981 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.363946 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.364358 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.364299 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.365753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.365727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.366724 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.366699 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.372595 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.372566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.372977 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.372956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzhn\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8dg8l\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.459776 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:35.459744 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436f87bc_5760_4817_bcf2_2cbbf329739a.slice/crio-26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551 WatchSource:0}: Error finding container 26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551: Status 404 returned error can't find the container with id 26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551 Nov 01 21:59:35.463032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.463006 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt"] Nov 01 21:59:35.506393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.506370 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:35.620903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.620820 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" event={"ID":"721797e9-5982-4f73-83f8-5d92ff41d904","Type":"ContainerStarted","Data":"86eac198a903c7f31003510fae1ce7c2b76612d5486ae75fecdc3f0c91d9d9e1"} Nov 01 21:59:35.620903 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.620896 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" event={"ID":"721797e9-5982-4f73-83f8-5d92ff41d904","Type":"ContainerStarted","Data":"318630abbae1365a136ee4f8950904d4a8c190d766d66e00464aa348f7c5304b"} Nov 01 21:59:35.622565 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.622534 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" event={"ID":"0608bb01-0084-4669-bb34-a2827102decf","Type":"ContainerStarted","Data":"1099e0bb756e3c0119f55786319ce3f43186b5878c3faafd81365565a07948f4"} Nov 01 21:59:35.624153 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.624129 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" event={"ID":"436f87bc-5760-4817-bcf2-2cbbf329739a","Type":"ContainerStarted","Data":"26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551"} Nov 01 21:59:35.648606 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.648553 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" podStartSLOduration=5.648536684 podStartE2EDuration="5.648536684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:35.638745754 +0000 UTC m=+929.331726240" watchObservedRunningTime="2026-11-01 21:59:35.648536684 +0000 UTC m=+929.341517182" Nov 01 21:59:35.896167 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:35.895963 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:36.057279 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.057246 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 21:59:36.061537 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:36.059893 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d01a05b_4fec_4b0d_963a_2d8987b5351a.slice/crio-0be2e1c075598e5a9a5d90890fe6d73f67d31b941c313d4c1303577190c21ddf WatchSource:0}: Error finding container 0be2e1c075598e5a9a5d90890fe6d73f67d31b941c313d4c1303577190c21ddf: Status 404 returned error can't find the container with id 0be2e1c075598e5a9a5d90890fe6d73f67d31b941c313d4c1303577190c21ddf Nov 01 21:59:36.338073 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.338049 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:36.636276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.636187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" event={"ID":"2d01a05b-4fec-4b0d-963a-2d8987b5351a","Type":"ContainerStarted","Data":"0be2e1c075598e5a9a5d90890fe6d73f67d31b941c313d4c1303577190c21ddf"} Nov 01 21:59:36.637964 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.637935 5036 generic.go:356] "Generic (PLEG): container finished" podID="721797e9-5982-4f73-83f8-5d92ff41d904" containerID="86eac198a903c7f31003510fae1ce7c2b76612d5486ae75fecdc3f0c91d9d9e1" exitCode=0 Nov 01 21:59:36.638103 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.638017 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" event={"ID":"721797e9-5982-4f73-83f8-5d92ff41d904","Type":"ContainerDied","Data":"86eac198a903c7f31003510fae1ce7c2b76612d5486ae75fecdc3f0c91d9d9e1"} Nov 01 21:59:36.639710 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.639684 5036 generic.go:356] "Generic (PLEG): container finished" podID="0608bb01-0084-4669-bb34-a2827102decf" containerID="5c0e96d6fcf5f016e777745224fc230936b7474a88ae7d752ce84424d56d260f" exitCode=0 Nov 01 21:59:36.639836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.639765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" event={"ID":"0608bb01-0084-4669-bb34-a2827102decf","Type":"ContainerDied","Data":"5c0e96d6fcf5f016e777745224fc230936b7474a88ae7d752ce84424d56d260f"} Nov 01 21:59:36.641487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.641430 5036 generic.go:356] "Generic (PLEG): container finished" podID="436f87bc-5760-4817-bcf2-2cbbf329739a" containerID="27edddef98d6dbf1081f2476e2a5c954f51ea34e34580cbfbff20b7b43d9f545" exitCode=0 Nov 01 21:59:36.641487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.641482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" event={"ID":"436f87bc-5760-4817-bcf2-2cbbf329739a","Type":"ContainerDied","Data":"27edddef98d6dbf1081f2476e2a5c954f51ea34e34580cbfbff20b7b43d9f545"} Nov 01 21:59:36.848382 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:36.848339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:37.625095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:37.625019 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:37.952867 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:37.951752 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:38.026211 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.025850 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\" (UniqueName: \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\") pod \"721797e9-5982-4f73-83f8-5d92ff41d904\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " Nov 01 21:59:38.027276 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.026864 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\") pod \"721797e9-5982-4f73-83f8-5d92ff41d904\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " Nov 01 21:59:38.027487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.027233 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca" pod "721797e9-5982-4f73-83f8-5d92ff41d904" (UID: "721797e9-5982-4f73-83f8-5d92ff41d904"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:38.027694 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.027667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\") pod \"721797e9-5982-4f73-83f8-5d92ff41d904\" (UID: \"721797e9-5982-4f73-83f8-5d92ff41d904\") " Nov 01 21:59:38.030070 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.029937 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln" pod "721797e9-5982-4f73-83f8-5d92ff41d904" (UID: "721797e9-5982-4f73-83f8-5d92ff41d904"). InnerVolumeSpecName "kube-api-access-qsxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:38.030999 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.030976 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxln\" (UniqueName: \"kubernetes.io/projected/721797e9-5982-4f73-83f8-5d92ff41d904-kube-api-access-qsxln\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.031104 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.031002 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.039573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.039550 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca" pod "721797e9-5982-4f73-83f8-5d92ff41d904" (UID: "721797e9-5982-4f73-83f8-5d92ff41d904"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:38.078483 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.076110 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:38.081393 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.080909 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:38.131939 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.131900 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\" (UniqueName: \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\") pod \"436f87bc-5760-4817-bcf2-2cbbf329739a\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " Nov 01 21:59:38.132186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.132050 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\") pod \"0608bb01-0084-4669-bb34-a2827102decf\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " Nov 01 21:59:38.132186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.132130 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\") pod \"436f87bc-5760-4817-bcf2-2cbbf329739a\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.132271 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\") pod \"0608bb01-0084-4669-bb34-a2827102decf\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.132417 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\" (UniqueName: \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\") pod \"0608bb01-0084-4669-bb34-a2827102decf\" (UID: \"0608bb01-0084-4669-bb34-a2827102decf\") " Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.132842 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\") pod \"436f87bc-5760-4817-bcf2-2cbbf329739a\" (UID: \"436f87bc-5760-4817-bcf2-2cbbf329739a\") " Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.133648 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721797e9-5982-4f73-83f8-5d92ff41d904-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.133992 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca" pod "436f87bc-5760-4817-bcf2-2cbbf329739a" (UID: "436f87bc-5760-4817-bcf2-2cbbf329739a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:38.137203 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.134894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca" pod "0608bb01-0084-4669-bb34-a2827102decf" (UID: "0608bb01-0084-4669-bb34-a2827102decf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:38.143032 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.143008 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct" pod "436f87bc-5760-4817-bcf2-2cbbf329739a" (UID: "436f87bc-5760-4817-bcf2-2cbbf329739a"). InnerVolumeSpecName "kube-api-access-kc9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:38.143177 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.143125 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca" pod "436f87bc-5760-4817-bcf2-2cbbf329739a" (UID: "436f87bc-5760-4817-bcf2-2cbbf329739a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:38.143254 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.143234 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca" pod "0608bb01-0084-4669-bb34-a2827102decf" (UID: "0608bb01-0084-4669-bb34-a2827102decf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:38.144746 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.144717 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8" pod "0608bb01-0084-4669-bb34-a2827102decf" (UID: "0608bb01-0084-4669-bb34-a2827102decf"). InnerVolumeSpecName "kube-api-access-cxbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:38.234779 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234743 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.234779 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234780 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9ct\" (UniqueName: \"kubernetes.io/projected/436f87bc-5760-4817-bcf2-2cbbf329739a-kube-api-access-kc9ct\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.234995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234791 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0608bb01-0084-4669-bb34-a2827102decf-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.234995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234801 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436f87bc-5760-4817-bcf2-2cbbf329739a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.234995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234815 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0608bb01-0084-4669-bb34-a2827102decf-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.234995 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.234829 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbb8\" (UniqueName: \"kubernetes.io/projected/0608bb01-0084-4669-bb34-a2827102decf-kube-api-access-cxbb8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:38.451631 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.451527 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:38.669669 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.669636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" event={"ID":"436f87bc-5760-4817-bcf2-2cbbf329739a","Type":"ContainerDied","Data":"26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551"} Nov 01 21:59:38.669863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.669730 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a56c08aa4ed98af81a0defceaddd89444a2852536c8cba25b2c06e5c9eb551" Nov 01 21:59:38.669863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.669810 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt" Nov 01 21:59:38.673054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.672407 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" Nov 01 21:59:38.673054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.672423 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd" event={"ID":"721797e9-5982-4f73-83f8-5d92ff41d904","Type":"ContainerDied","Data":"318630abbae1365a136ee4f8950904d4a8c190d766d66e00464aa348f7c5304b"} Nov 01 21:59:38.673054 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.672509 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="318630abbae1365a136ee4f8950904d4a8c190d766d66e00464aa348f7c5304b" Nov 01 21:59:38.675228 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.675201 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" event={"ID":"0608bb01-0084-4669-bb34-a2827102decf","Type":"ContainerDied","Data":"1099e0bb756e3c0119f55786319ce3f43186b5878c3faafd81365565a07948f4"} Nov 01 21:59:38.675339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.675249 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1099e0bb756e3c0119f55786319ce3f43186b5878c3faafd81365565a07948f4" Nov 01 21:59:38.675339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:38.675301 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z" Nov 01 21:59:39.233559 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:39.233513 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:39.978300 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:39.978266 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:39.978428 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:39.978348 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 21:59:40.051699 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.051675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:40.558137 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.558098 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:40.639584 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.638700 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:40.716494 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.715955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" event={"ID":"2d01a05b-4fec-4b0d-963a-2d8987b5351a","Type":"ContainerStarted","Data":"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735"} Nov 01 21:59:40.717306 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.717124 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:40.747401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.747340 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" podStartSLOduration=4.832643972 podStartE2EDuration="8.747320698s" totalImagesPullingTime="3.914676726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:40.742496474 +0000 UTC m=+934.435476956" watchObservedRunningTime="2026-11-01 21:59:40.747320698 +0000 UTC m=+934.440301191" Nov 01 21:59:40.767169 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.767142 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:40.978145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:40.978056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:41.103753 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.103713 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:41.216467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.190347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:41.367185 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.364411 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:41.743062 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.742969 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 21:59:41.964550 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.963342 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:41.967733 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.967150 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:41.990544 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:41.987707 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:42.101635 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.101519 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:42.282489 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.280976 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:42.576093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.576055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:42.708472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.704827 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:42.732654 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.732618 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt"] Nov 01 21:59:42.739748 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.739718 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93a347eb-stdlt"] Nov 01 21:59:42.930645 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:42.930560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:43.714405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:43.714361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:43.853249 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:43.853210 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:43.985528 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:43.981474 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:44.551475 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:44.550735 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436f87bc-5760-4817-bcf2-2cbbf329739a" path="/var/lib/kubelet/pods/436f87bc-5760-4817-bcf2-2cbbf329739a/volumes" Nov 01 21:59:44.619157 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:44.619105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:44.860100 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:44.860017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:45.559339 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:45.559297 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" podUID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" containerName="payload-pre-processing" containerID="cri-o://7f48e919e1fdc7f625c0da883daf54245c2ab81e81e474b0347e086a224b59bd" gracePeriod=25 Nov 01 21:59:45.623116 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:45.623084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:45.805145 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:45.805113 5036 generic.go:356] "Generic (PLEG): container finished" podID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" containerID="7f48e919e1fdc7f625c0da883daf54245c2ab81e81e474b0347e086a224b59bd" exitCode=0 Nov 01 21:59:45.805601 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:45.805578 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" event={"ID":"068c06a1-d6ac-4c39-95e6-792c876ffa6d","Type":"ContainerDied","Data":"7f48e919e1fdc7f625c0da883daf54245c2ab81e81e474b0347e086a224b59bd"} Nov 01 21:59:45.902741 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:45.902714 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:46.034933 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.034896 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\" (UniqueName: \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\") pod \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " Nov 01 21:59:46.035174 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.035153 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\") pod \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\" (UID: \"068c06a1-d6ac-4c39-95e6-792c876ffa6d\") " Nov 01 21:59:46.035698 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.035667 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume" pod "068c06a1-d6ac-4c39-95e6-792c876ffa6d" (UID: "068c06a1-d6ac-4c39-95e6-792c876ffa6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:46.035836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.035801 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068c06a1-d6ac-4c39-95e6-792c876ffa6d-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.037426 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.037393 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l" pod "068c06a1-d6ac-4c39-95e6-792c876ffa6d" (UID: "068c06a1-d6ac-4c39-95e6-792c876ffa6d"). InnerVolumeSpecName "kube-api-access-m5p6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:46.104453 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.104375 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" podUID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" containerName="payload-pre-processing" containerID="cri-o://a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338" gracePeriod=25 Nov 01 21:59:46.136905 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.136880 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5p6l\" (UniqueName: \"kubernetes.io/projected/068c06a1-d6ac-4c39-95e6-792c876ffa6d-kube-api-access-m5p6l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.342468 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.341972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:46.370774 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.370744 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:46.443944 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.443910 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\" (UniqueName: \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\") pod \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " Nov 01 21:59:46.444246 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.444224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\") pod \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\" (UID: \"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac\") " Nov 01 21:59:46.444839 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.444807 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume" pod "f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" (UID: "f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:46.447306 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.447278 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.451771 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.451742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc" pod "f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" (UID: "f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac"). InnerVolumeSpecName "kube-api-access-j5zpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:46.548625 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.548590 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zpc\" (UniqueName: \"kubernetes.io/projected/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac-kube-api-access-j5zpc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.597025 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.596974 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:46.816512 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.816486 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" Nov 01 21:59:46.816714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.816485 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr" event={"ID":"068c06a1-d6ac-4c39-95e6-792c876ffa6d","Type":"ContainerDied","Data":"47b965b85c976b9c2097ea7d044a919213bccf2f103f747ad5ecf3da48417701"} Nov 01 21:59:46.816714 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.816626 5036 scope.go:122] "RemoveContainer" containerID="7f48e919e1fdc7f625c0da883daf54245c2ab81e81e474b0347e086a224b59bd" Nov 01 21:59:46.818164 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.818144 5036 generic.go:356] "Generic (PLEG): container finished" podID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" containerID="a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338" exitCode=0 Nov 01 21:59:46.818263 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.818198 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" Nov 01 21:59:46.818263 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.818217 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" event={"ID":"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac","Type":"ContainerDied","Data":"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338"} Nov 01 21:59:46.818351 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.818268 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7" event={"ID":"f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac","Type":"ContainerDied","Data":"ed451909365eb9dea9a688d49a48318f105ac81016cc4120a0d6c45e4372d368"} Nov 01 21:59:46.829271 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.829250 5036 scope.go:122] "RemoveContainer" containerID="a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338" Nov 01 21:59:46.842936 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.842916 5036 scope.go:122] "RemoveContainer" containerID="a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338" Nov 01 21:59:46.843236 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:46.843213 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338\": container with ID starting with a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338 not found: ID does not exist" containerID="a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338" Nov 01 21:59:46.843299 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.843249 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338"} err="failed to get container status \"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338\": rpc error: code = NotFound desc = could not find container \"a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338\": container with ID starting with a9507deb978a6de0ab41d7d38aca2cdaa213ea3dc39ff2897ee119cea5d20338 not found: ID does not exist" Nov 01 21:59:46.895227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.895201 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:46.903173 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.903148 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681db477-58c4bb787-pnsjr"] Nov 01 21:59:46.915721 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.915693 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:46.923431 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.923407 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13364262-55c96d8b7f-mktk7"] Nov 01 21:59:46.964711 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.964683 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" podUID="ef46ac92-7da8-47ba-9a8b-091fb244558d" containerName="payload-processing" containerID="cri-o://031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347" gracePeriod=25 Nov 01 21:59:46.988144 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:46.988110 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" podUID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" containerName="payload-pre-processing" containerID="cri-o://9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a" gracePeriod=25 Nov 01 21:59:47.145963 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.145837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:47.345150 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.345074 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:47.393026 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.392658 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:47.473095 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.473024 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\") pod \"ef46ac92-7da8-47ba-9a8b-091fb244558d\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " Nov 01 21:59:47.473252 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.473127 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\" (UniqueName: \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\") pod \"ef46ac92-7da8-47ba-9a8b-091fb244558d\" (UID: \"ef46ac92-7da8-47ba-9a8b-091fb244558d\") " Nov 01 21:59:47.473521 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.473493 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume" pod "ef46ac92-7da8-47ba-9a8b-091fb244558d" (UID: "ef46ac92-7da8-47ba-9a8b-091fb244558d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:47.473845 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.473827 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef46ac92-7da8-47ba-9a8b-091fb244558d-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:47.475294 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.475269 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn" pod "ef46ac92-7da8-47ba-9a8b-091fb244558d" (UID: "ef46ac92-7da8-47ba-9a8b-091fb244558d"). InnerVolumeSpecName "kube-api-access-tc4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:47.574978 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.574948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\") pod \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " Nov 01 21:59:47.575129 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.575087 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\" (UniqueName: \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\") pod \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\" (UID: \"17afae5b-e30e-4d5f-93ea-4b03010a5cfa\") " Nov 01 21:59:47.575385 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.575357 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume" pod "17afae5b-e30e-4d5f-93ea-4b03010a5cfa" (UID: "17afae5b-e30e-4d5f-93ea-4b03010a5cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:47.575804 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.575781 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4zn\" (UniqueName: \"kubernetes.io/projected/ef46ac92-7da8-47ba-9a8b-091fb244558d-kube-api-access-tc4zn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:47.575994 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.575972 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:47.577422 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.577397 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2" pod "17afae5b-e30e-4d5f-93ea-4b03010a5cfa" (UID: "17afae5b-e30e-4d5f-93ea-4b03010a5cfa"). InnerVolumeSpecName "kube-api-access-fc2m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:47.677093 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.677024 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc2m2\" (UniqueName: \"kubernetes.io/projected/17afae5b-e30e-4d5f-93ea-4b03010a5cfa-kube-api-access-fc2m2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:47.827039 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.827000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:47.830482 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.830415 5036 generic.go:356] "Generic (PLEG): container finished" podID="ef46ac92-7da8-47ba-9a8b-091fb244558d" containerID="031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347" exitCode=0 Nov 01 21:59:47.830691 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.830662 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" event={"ID":"ef46ac92-7da8-47ba-9a8b-091fb244558d","Type":"ContainerDied","Data":"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347"} Nov 01 21:59:47.830820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.830723 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" event={"ID":"ef46ac92-7da8-47ba-9a8b-091fb244558d","Type":"ContainerDied","Data":"eed9338a8df1a79369c13fce38263f269b54af81d9baca4526a1290d3cea0801"} Nov 01 21:59:47.830820 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.830767 5036 scope.go:122] "RemoveContainer" containerID="031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347" Nov 01 21:59:47.831021 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.831007 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9" Nov 01 21:59:47.842886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.842558 5036 generic.go:356] "Generic (PLEG): container finished" podID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" containerID="9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a" exitCode=0 Nov 01 21:59:47.842886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.842617 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" Nov 01 21:59:47.842886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.842637 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" event={"ID":"17afae5b-e30e-4d5f-93ea-4b03010a5cfa","Type":"ContainerDied","Data":"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a"} Nov 01 21:59:47.842886 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.842683 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm" event={"ID":"17afae5b-e30e-4d5f-93ea-4b03010a5cfa","Type":"ContainerDied","Data":"a4641a650b1f891e4a40ceaf5d058aca2ce0f26c527ed7b2892b87919b2889bb"} Nov 01 21:59:47.862227 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.862200 5036 scope.go:122] "RemoveContainer" containerID="031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347" Nov 01 21:59:47.862924 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:47.862898 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347\": container with ID starting with 031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347 not found: ID does not exist" containerID="031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347" Nov 01 21:59:47.863052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.862934 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347"} err="failed to get container status \"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347\": rpc error: code = NotFound desc = could not find container \"031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347\": container with ID starting with 031b19bbcf7713a52566165671205abb26aee35564f09ac2547582a7b9975347 not found: ID does not exist" Nov 01 21:59:47.863052 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.862960 5036 scope.go:122] "RemoveContainer" containerID="9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a" Nov 01 21:59:47.872289 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.872249 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:47.890719 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.890615 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93a347eb-67ff94dc58-f7rb9"] Nov 01 21:59:47.890719 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.890640 5036 scope.go:122] "RemoveContainer" containerID="9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a" Nov 01 21:59:47.891686 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:47.891662 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a\": container with ID starting with 9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a not found: ID does not exist" containerID="9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a" Nov 01 21:59:47.891807 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.891696 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a"} err="failed to get container status \"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a\": rpc error: code = NotFound desc = could not find container \"9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a\": container with ID starting with 9dbf8d7d728463835a5becd221a85cbfc9ff519d98b3c0f05919949cfc93f71a not found: ID does not exist" Nov 01 21:59:47.898143 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.898091 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:47.907375 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:47.907355 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93a347eb-d9b77546c-qdnhm"] Nov 01 21:59:48.440836 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:48.440798 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" path="/var/lib/kubelet/pods/068c06a1-d6ac-4c39-95e6-792c876ffa6d/volumes" Nov 01 21:59:48.441256 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:48.441240 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" path="/var/lib/kubelet/pods/17afae5b-e30e-4d5f-93ea-4b03010a5cfa/volumes" Nov 01 21:59:48.441680 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:48.441663 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef46ac92-7da8-47ba-9a8b-091fb244558d" path="/var/lib/kubelet/pods/ef46ac92-7da8-47ba-9a8b-091fb244558d/volumes" Nov 01 21:59:48.442049 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:48.442035 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" path="/var/lib/kubelet/pods/f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac/volumes" Nov 01 21:59:48.529315 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:48.529279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:48.860992 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:48.860959 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 21:59:49.107386 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.107352 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:49.267411 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.267320 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:49.287685 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.287649 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd"] Nov 01 21:59:49.294156 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.294130 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13364262-b2pbd"] Nov 01 21:59:49.404508 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.404221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:49.778519 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.777392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:49.824105 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.824072 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z"] Nov 01 21:59:49.830504 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.830477 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-681db477-8nb7z"] Nov 01 21:59:49.909980 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:49.909947 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:50.036765 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.036681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:50.459420 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.459325 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0608bb01-0084-4669-bb34-a2827102decf" path="/var/lib/kubelet/pods/0608bb01-0084-4669-bb34-a2827102decf/volumes" Nov 01 21:59:50.459985 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.459962 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721797e9-5982-4f73-83f8-5d92ff41d904" path="/var/lib/kubelet/pods/721797e9-5982-4f73-83f8-5d92ff41d904/volumes" Nov 01 21:59:50.471147 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.471118 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:50.471377 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.471329 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" podUID="9d936208-b896-4a9c-a225-7551d3902f49" containerName="istio-proxy" containerID="cri-o://8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc" gracePeriod=30 Nov 01 21:59:50.501165 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.501126 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:50.840920 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:50.840875 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:51.125517 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:51.116252 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:51.432231 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:51.432151 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:51.655467 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:51.649181 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:51.797935 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:51.797900 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:51.958943 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:51.958905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:52.768469 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:52.763199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:53.642413 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:53.642367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:53.911661 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:53.911569 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:54.103913 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:54.103878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:54.235186 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:54.235094 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:54.545505 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:54.545465 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:55.303569 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.303533 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:55.436573 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.436538 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:55.886725 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.886701 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:55.967401 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.967366 5036 generic.go:356] "Generic (PLEG): container finished" podID="9d936208-b896-4a9c-a225-7551d3902f49" containerID="8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc" exitCode=0 Nov 01 21:59:55.967625 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.967416 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" Nov 01 21:59:55.967625 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.967487 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" event={"ID":"9d936208-b896-4a9c-a225-7551d3902f49","Type":"ContainerDied","Data":"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc"} Nov 01 21:59:55.967625 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.967555 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9" event={"ID":"9d936208-b896-4a9c-a225-7551d3902f49","Type":"ContainerDied","Data":"5951bcb4a4f55c362dc9b74b166293236d781c491ffa912e1e377d572b8ff84e"} Nov 01 21:59:55.967625 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.967588 5036 scope.go:122] "RemoveContainer" containerID="8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc" Nov 01 21:59:55.980046 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.980025 5036 scope.go:122] "RemoveContainer" containerID="8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc" Nov 01 21:59:55.980344 ip-10-0-131-110 kubenswrapper[5036]: E1101 21:59:55.980324 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc\": container with ID starting with 8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc not found: ID does not exist" containerID="8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc" Nov 01 21:59:55.980438 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:55.980353 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc"} err="failed to get container status \"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc\": rpc error: code = NotFound desc = could not find container \"8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc\": container with ID starting with 8225d3e22ba524ebd34035119377ae3a1979659252c3b7e2d7b94216095e9fdc not found: ID does not exist" Nov 01 21:59:56.041962 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.041930 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042088 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.041984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042088 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042049 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042200 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042173 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042200 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:56.042262 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042209 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042405 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042361 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042487 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042432 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042554 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:56.042616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042551 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042616 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\") pod \"9d936208-b896-4a9c-a225-7551d3902f49\" (UID: \"9d936208-b896-4a9c-a225-7551d3902f49\") " Nov 01 21:59:56.042823 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042783 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:56.042889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042843 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:56.042889 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.042858 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:56.043328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.043306 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.043328 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.043328 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.043491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.043339 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.043491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.043365 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d936208-b896-4a9c-a225-7551d3902f49-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.043491 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.043374 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.044712 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.044687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:59:56.044793 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.044695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:56.044863 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.044842 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:56.044897 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.044846 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq" pod "9d936208-b896-4a9c-a225-7551d3902f49" (UID: "9d936208-b896-4a9c-a225-7551d3902f49"). InnerVolumeSpecName "kube-api-access-6xndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:56.144563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.144490 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xndq\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-kube-api-access-6xndq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.144563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.144521 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d936208-b896-4a9c-a225-7551d3902f49-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.144563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.144532 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d936208-b896-4a9c-a225-7551d3902f49-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.144563 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.144544 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d936208-b896-4a9c-a225-7551d3902f49-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:56.298966 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.298933 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:56.305096 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.305071 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-681db477-openshift-default-74959dc87f-fddv9"] Nov 01 21:59:56.340778 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.340741 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:56.443007 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:56.442919 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d936208-b896-4a9c-a225-7551d3902f49" path="/var/lib/kubelet/pods/9d936208-b896-4a9c-a225-7551d3902f49/volumes" Nov 01 21:59:57.113648 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:57.113581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:57.248894 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:57.248855 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:57.937940 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:57.937902 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:58.633986 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.633947 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:58.915884 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.915804 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr"] Nov 01 21:59:58.916874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916850 5036 cpu_manager.go:414] "containerMap: removing container" podUID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" containerName="payload-pre-processing" Nov 01 21:59:58.916874 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916872 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" containerName="payload-pre-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916892 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9d936208-b896-4a9c-a225-7551d3902f49" containerName="istio-proxy" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916900 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d936208-b896-4a9c-a225-7551d3902f49" containerName="istio-proxy" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916920 5036 cpu_manager.go:414] "containerMap: removing container" podUID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" containerName="payload-pre-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916928 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" containerName="payload-pre-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916986 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ef46ac92-7da8-47ba-9a8b-091fb244558d" containerName="payload-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.916994 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef46ac92-7da8-47ba-9a8b-091fb244558d" containerName="payload-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917006 5036 cpu_manager.go:414] "containerMap: removing container" podUID="721797e9-5982-4f73-83f8-5d92ff41d904" containerName="revoke-keys" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917013 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721797e9-5982-4f73-83f8-5d92ff41d904" containerName="revoke-keys" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917027 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" containerName="payload-pre-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917034 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" containerName="payload-pre-processing" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917045 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0608bb01-0084-4669-bb34-a2827102decf" containerName="revoke-keys" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917051 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0608bb01-0084-4669-bb34-a2827102decf" containerName="revoke-keys" Nov 01 21:59:58.917063 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917064 5036 cpu_manager.go:414] "containerMap: removing container" podUID="436f87bc-5760-4817-bcf2-2cbbf329739a" containerName="revoke-keys" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917072 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436f87bc-5760-4817-bcf2-2cbbf329739a" containerName="revoke-keys" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917217 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef46ac92-7da8-47ba-9a8b-091fb244558d" containerName="payload-processing" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917228 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="721797e9-5982-4f73-83f8-5d92ff41d904" containerName="revoke-keys" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917238 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ccc8dd-8eef-4eb5-8234-a1749b2c4bac" containerName="payload-pre-processing" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917249 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0608bb01-0084-4669-bb34-a2827102decf" containerName="revoke-keys" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917259 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="436f87bc-5760-4817-bcf2-2cbbf329739a" containerName="revoke-keys" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917271 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="17afae5b-e30e-4d5f-93ea-4b03010a5cfa" containerName="payload-pre-processing" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917280 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d936208-b896-4a9c-a225-7551d3902f49" containerName="istio-proxy" Nov 01 21:59:58.917472 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.917292 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="068c06a1-d6ac-4c39-95e6-792c876ffa6d" containerName="payload-pre-processing" Nov 01 21:59:58.926197 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.926171 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:58.926511 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.926485 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr"] Nov 01 21:59:58.932245 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.932222 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:59:58.933580 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.933552 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:59:58.933706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.933655 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:59:58.933706 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:58.933682 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:59:59.089578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.089292 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.089578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.089373 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.089578 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.089429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr9k\" (UniqueName: \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.190541 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.190425 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.190707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.190542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.190707 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.190614 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr9k\" (UniqueName: \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.191260 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.191226 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.192976 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.192950 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.202651 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.202630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr9k\" (UniqueName: \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\") pod \"maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.247027 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.246994 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 21:59:59.446605 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.446476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:59.534542 ip-10-0-131-110 kubenswrapper[5036]: W1101 21:59:59.534510 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24c62fc_0e0b_45a8_a64a_e2bdedeb63a6.slice/crio-70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535 WatchSource:0}: Error finding container 70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535: Status 404 returned error can't find the container with id 70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535 Nov 01 21:59:59.537911 ip-10-0-131-110 kubenswrapper[5036]: I1101 21:59:59.537882 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr"] Nov 01 22:00:00.051984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.051950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" event={"ID":"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6","Type":"ContainerStarted","Data":"6257654dbc72a638f30f59009aacdf5c92fbcba845ee629fe1e4abadc79705c6"} Nov 01 22:00:00.052203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.052008 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" event={"ID":"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6","Type":"ContainerStarted","Data":"70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535"} Nov 01 22:00:00.074357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.074303 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" podStartSLOduration=2.074287193 podStartE2EDuration="2.074287193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:00.071342901 +0000 UTC m=+953.764323388" watchObservedRunningTime="2026-11-01 22:00:00.074287193 +0000 UTC m=+953.767267678" Nov 01 22:00:00.149837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.149756 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:00.155189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.155165 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.157620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.157595 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:00.167436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.167411 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:00.170208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.170184 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.183191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.181168 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.192216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.189753 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:00.199900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.199855 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:00.211994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.211965 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:00.223398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.223374 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.224697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.224677 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:00.239063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.239045 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:00.254222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.254199 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.262163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.262128 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:00.267103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.267077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:00.312371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312304 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.312371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312366 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4k5\" (UniqueName: \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.312630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312395 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.312630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312454 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.312630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2wt\" (UniqueName: \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.312800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrwn\" (UniqueName: \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.312800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312761 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.312906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.312797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tws6q\" (UniqueName: \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.314307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.314279 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:00.414503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414349 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.414503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414406 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4k5\" (UniqueName: \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.414503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414431 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.414817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.414817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414588 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.414817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414658 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2wt\" (UniqueName: \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.414817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414689 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwxg\" (UniqueName: \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.414817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.414738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrwn\" (UniqueName: \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.415104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.415028 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.415104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.415074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tws6q\" (UniqueName: \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.415211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.415176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.415211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.415179 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.415858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.415830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.416041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.416019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.428030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.427975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrwn\" (UniqueName: \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\") pod \"maas-api-key-cleanup-29892840-s7996\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.428158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.428130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4k5\" (UniqueName: \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.428366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.428344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tws6q\" (UniqueName: \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.429151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.429122 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2wt\" (UniqueName: \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.499414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.499386 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:00.517416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.517268 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.517996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.517971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwxg\" (UniqueName: \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.518160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.518135 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.530131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.530111 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwxg\" (UniqueName: \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\") pod \"maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.545173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.545148 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:00.573853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.573830 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:00.637709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.635256 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:00.638968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.638279 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:00.816233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.815976 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:00.866556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.866526 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:00.925710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.925641 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:00.958455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.958389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:00.968853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:00.968829 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:01.009761 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:00:01.009736 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5f03cc_4d69_4383_b16f_abb324a0f59a.slice/crio-78772ed0c1599666f5240bfaa91b13f5e88bfded0b42b0336a280123939886c7 WatchSource:0}: Error finding container 78772ed0c1599666f5240bfaa91b13f5e88bfded0b42b0336a280123939886c7: Status 404 returned error can't find the container with id 78772ed0c1599666f5240bfaa91b13f5e88bfded0b42b0336a280123939886c7 Nov 01 22:00:01.012753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.012729 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:01.062670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.062646 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerStarted","Data":"e770e0e600a14e3246db5bb1f3f5e82703b4a0a332ea368e4bfedcf5306ac69b"} Nov 01 22:00:01.064836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.064812 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerStarted","Data":"92c5ece31a42e1c6b2505c45ca4a3004eddb9ae2dcfcaf2c69e42c03ed4813ba"} Nov 01 22:00:01.066419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.066395 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerStarted","Data":"a4938a631e3e24dd3be65251156c574dc593b1ac4d889cab9fe5d598f9c012d9"} Nov 01 22:00:01.071928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.071898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerStarted","Data":"72f2a3e5dea49a5748598cc4426c82d5a07d019cb89637b3f05adc8fbab63c2c"} Nov 01 22:00:01.073938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.073897 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerStarted","Data":"78772ed0c1599666f5240bfaa91b13f5e88bfded0b42b0336a280123939886c7"} Nov 01 22:00:01.087576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.086954 5036 generic.go:356] "Generic (PLEG): container finished" podID="a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" containerID="6257654dbc72a638f30f59009aacdf5c92fbcba845ee629fe1e4abadc79705c6" exitCode=0 Nov 01 22:00:01.087576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.087022 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" event={"ID":"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6","Type":"ContainerDied","Data":"6257654dbc72a638f30f59009aacdf5c92fbcba845ee629fe1e4abadc79705c6"} Nov 01 22:00:01.105080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.105042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:01.929516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:01.929378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:02.104753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.104724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerStarted","Data":"15ba884d42fe72036678d4c2d02bdf01025e8b84c6c7aa89fd275c26aacb4df3"} Nov 01 22:00:02.106959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.106932 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerID="4368ecb40c49cc79b2d59ebf27e19269767a97cfd26c73e4d66686a0c877f76c" exitCode=60 Nov 01 22:00:02.107081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.106962 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerDied","Data":"4368ecb40c49cc79b2d59ebf27e19269767a97cfd26c73e4d66686a0c877f76c"} Nov 01 22:00:02.107356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.107338 5036 scope.go:122] "RemoveContainer" containerID="4368ecb40c49cc79b2d59ebf27e19269767a97cfd26c73e4d66686a0c877f76c" Nov 01 22:00:02.109467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.109104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerStarted","Data":"1ce4809fd1e514bdaa4d7013a6cb382152a401157b131a7d57bda1d95ffc3e7a"} Nov 01 22:00:02.109467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.109403 5036 scope.go:122] "RemoveContainer" containerID="1ce4809fd1e514bdaa4d7013a6cb382152a401157b131a7d57bda1d95ffc3e7a" Nov 01 22:00:02.113882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.113436 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerStarted","Data":"1c69ecd892b8c47c14a483c22d73d12adf5e2ae62f6d6fb6ea31ae6a5f709e5a"} Nov 01 22:00:02.113882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.113751 5036 scope.go:122] "RemoveContainer" containerID="1c69ecd892b8c47c14a483c22d73d12adf5e2ae62f6d6fb6ea31ae6a5f709e5a" Nov 01 22:00:02.116525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.116503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerStarted","Data":"73fc38dbacca8346055fc605b7b47cbc5622006ae2b9d3e8ec3f85424ba902a2"} Nov 01 22:00:02.123854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.123796 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" podStartSLOduration=2.123780274 podStartE2EDuration="2.123780274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:02.122870527 +0000 UTC m=+955.815851013" watchObservedRunningTime="2026-11-01 22:00:02.123780274 +0000 UTC m=+955.816760761" Nov 01 22:00:02.362075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.362049 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 22:00:02.453675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.453553 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" podStartSLOduration=2.453535793 podStartE2EDuration="2.453535793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:02.23167053 +0000 UTC m=+955.924651016" watchObservedRunningTime="2026-11-01 22:00:02.453535793 +0000 UTC m=+956.146516280" Nov 01 22:00:02.468572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.468540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\") pod \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " Nov 01 22:00:02.468825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.468806 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\" (UniqueName: \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\") pod \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " Nov 01 22:00:02.469349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.469321 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\") pod \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\" (UID: \"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6\") " Nov 01 22:00:02.469526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.469383 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca" pod "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" (UID: "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:02.470062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.470034 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:02.480045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.480000 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca" pod "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" (UID: "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:00:02.480187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.480065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k" pod "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" (UID: "a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6"). InnerVolumeSpecName "kube-api-access-dwr9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:02.570962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.570929 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwr9k\" (UniqueName: \"kubernetes.io/projected/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-kube-api-access-dwr9k\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:02.570962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:02.570959 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:03.010127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.010090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:03.161943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.160458 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerStarted","Data":"297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1"} Nov 01 22:00:03.169380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.168247 5036 generic.go:356] "Generic (PLEG): container finished" podID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerID="1ce4809fd1e514bdaa4d7013a6cb382152a401157b131a7d57bda1d95ffc3e7a" exitCode=60 Nov 01 22:00:03.169380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.168291 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerDied","Data":"1ce4809fd1e514bdaa4d7013a6cb382152a401157b131a7d57bda1d95ffc3e7a"} Nov 01 22:00:03.169380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.168525 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerStarted","Data":"9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2"} Nov 01 22:00:03.171102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.171068 5036 generic.go:356] "Generic (PLEG): container finished" podID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerID="1c69ecd892b8c47c14a483c22d73d12adf5e2ae62f6d6fb6ea31ae6a5f709e5a" exitCode=60 Nov 01 22:00:03.171235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.171139 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerDied","Data":"1c69ecd892b8c47c14a483c22d73d12adf5e2ae62f6d6fb6ea31ae6a5f709e5a"} Nov 01 22:00:03.171235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.171170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerStarted","Data":"6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22"} Nov 01 22:00:03.174711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.174688 5036 generic.go:356] "Generic (PLEG): container finished" podID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerID="73fc38dbacca8346055fc605b7b47cbc5622006ae2b9d3e8ec3f85424ba902a2" exitCode=60 Nov 01 22:00:03.176090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.176063 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerDied","Data":"73fc38dbacca8346055fc605b7b47cbc5622006ae2b9d3e8ec3f85424ba902a2"} Nov 01 22:00:03.177189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.176650 5036 scope.go:122] "RemoveContainer" containerID="73fc38dbacca8346055fc605b7b47cbc5622006ae2b9d3e8ec3f85424ba902a2" Nov 01 22:00:03.179075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.178756 5036 generic.go:356] "Generic (PLEG): container finished" podID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerID="15ba884d42fe72036678d4c2d02bdf01025e8b84c6c7aa89fd275c26aacb4df3" exitCode=60 Nov 01 22:00:03.179075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.178842 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerDied","Data":"15ba884d42fe72036678d4c2d02bdf01025e8b84c6c7aa89fd275c26aacb4df3"} Nov 01 22:00:03.179199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.179148 5036 scope.go:122] "RemoveContainer" containerID="15ba884d42fe72036678d4c2d02bdf01025e8b84c6c7aa89fd275c26aacb4df3" Nov 01 22:00:03.201682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.201655 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" event={"ID":"a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6","Type":"ContainerDied","Data":"70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535"} Nov 01 22:00:03.201840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.201823 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70179d3ed2dbcce7cccdd8eea0607d747b87fd40de8f8cb81ffca43c49311535" Nov 01 22:00:03.201996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.201984 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr" Nov 01 22:00:03.229708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.226714 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" podStartSLOduration=3.226698061 podStartE2EDuration="3.226698061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:03.198243472 +0000 UTC m=+956.891223958" watchObservedRunningTime="2026-11-01 22:00:03.226698061 +0000 UTC m=+956.919678550" Nov 01 22:00:03.230876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.230838 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" podStartSLOduration=3.230825604 podStartE2EDuration="3.230825604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:03.219220754 +0000 UTC m=+956.912201239" watchObservedRunningTime="2026-11-01 22:00:03.230825604 +0000 UTC m=+956.923806090" Nov 01 22:00:03.274009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.273964 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" podStartSLOduration=3.273952668 podStartE2EDuration="3.273952668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:03.272535729 +0000 UTC m=+956.965516215" watchObservedRunningTime="2026-11-01 22:00:03.273952668 +0000 UTC m=+956.966933154" Nov 01 22:00:03.820355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:03.820325 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:04.214630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.214528 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerStarted","Data":"f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51"} Nov 01 22:00:04.217072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.217043 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerID="297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1" exitCode=60 Nov 01 22:00:04.217230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.217123 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerDied","Data":"297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1"} Nov 01 22:00:04.217230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.217187 5036 scope.go:122] "RemoveContainer" containerID="4368ecb40c49cc79b2d59ebf27e19269767a97cfd26c73e4d66686a0c877f76c" Nov 01 22:00:04.217369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.217343 5036 scope.go:122] "RemoveContainer" containerID="297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1" Nov 01 22:00:04.217670 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:04.217645 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x_odh-ai-gateway-infra(6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" Nov 01 22:00:04.219616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.219597 5036 generic.go:356] "Generic (PLEG): container finished" podID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerID="9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2" exitCode=60 Nov 01 22:00:04.219720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.219677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerDied","Data":"9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2"} Nov 01 22:00:04.219885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.219863 5036 scope.go:122] "RemoveContainer" containerID="9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2" Nov 01 22:00:04.220090 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:04.220069 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c_odh-ai-gateway-infra(739c36a8-4f72-4837-93a0-b2d3f0402568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" Nov 01 22:00:04.221847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.221824 5036 generic.go:356] "Generic (PLEG): container finished" podID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerID="6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22" exitCode=60 Nov 01 22:00:04.221965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.221905 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerDied","Data":"6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22"} Nov 01 22:00:04.222099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.222083 5036 scope.go:122] "RemoveContainer" containerID="6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22" Nov 01 22:00:04.222321 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:04.222297 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-s7996_odh-ai-gateway-infra(7897dce5-113f-4f80-a61a-aa25e14bf232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" Nov 01 22:00:04.224199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.224176 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerStarted","Data":"1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce"} Nov 01 22:00:04.234862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.234846 5036 scope.go:122] "RemoveContainer" containerID="1ce4809fd1e514bdaa4d7013a6cb382152a401157b131a7d57bda1d95ffc3e7a" Nov 01 22:00:04.254619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.254583 5036 scope.go:122] "RemoveContainer" containerID="1c69ecd892b8c47c14a483c22d73d12adf5e2ae62f6d6fb6ea31ae6a5f709e5a" Nov 01 22:00:04.580695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.580640 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:04.892362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:04.892323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:00:05.264143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.264055 5036 scope.go:122] "RemoveContainer" containerID="297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1" Nov 01 22:00:05.264616 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:05.264502 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x_odh-ai-gateway-infra(6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" Nov 01 22:00:05.266350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.266325 5036 scope.go:122] "RemoveContainer" containerID="9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2" Nov 01 22:00:05.266689 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:05.266668 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c_odh-ai-gateway-infra(739c36a8-4f72-4837-93a0-b2d3f0402568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" Nov 01 22:00:05.268182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.268163 5036 scope.go:122] "RemoveContainer" containerID="6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22" Nov 01 22:00:05.268516 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:05.268492 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-s7996_odh-ai-gateway-infra(7897dce5-113f-4f80-a61a-aa25e14bf232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" Nov 01 22:00:05.270908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.270884 5036 generic.go:356] "Generic (PLEG): container finished" podID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerID="1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce" exitCode=60 Nov 01 22:00:05.271102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.271086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerDied","Data":"1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce"} Nov 01 22:00:05.271184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.271125 5036 scope.go:122] "RemoveContainer" containerID="73fc38dbacca8346055fc605b7b47cbc5622006ae2b9d3e8ec3f85424ba902a2" Nov 01 22:00:05.271236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.271212 5036 scope.go:122] "RemoveContainer" containerID="1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce" Nov 01 22:00:05.271504 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:05.271482 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l_odh-ai-gateway-infra(aed6254e-e206-408f-91a3-36ad5b4fbf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" Nov 01 22:00:05.273286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.273262 5036 generic.go:356] "Generic (PLEG): container finished" podID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerID="f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51" exitCode=60 Nov 01 22:00:05.273393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.273337 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerDied","Data":"f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51"} Nov 01 22:00:05.273649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.273633 5036 scope.go:122] "RemoveContainer" containerID="f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51" Nov 01 22:00:05.273886 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:05.273869 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd_odh-ai-gateway-infra(ae5f03cc-4d69-4383-b16f-abb324a0f59a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" Nov 01 22:00:05.289925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.289900 5036 scope.go:122] "RemoveContainer" containerID="15ba884d42fe72036678d4c2d02bdf01025e8b84c6c7aa89fd275c26aacb4df3" Nov 01 22:00:05.448227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:05.448182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:06.318469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:06.317609 5036 scope.go:122] "RemoveContainer" containerID="1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce" Nov 01 22:00:06.318469 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:06.318018 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l_odh-ai-gateway-infra(aed6254e-e206-408f-91a3-36ad5b4fbf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" Nov 01 22:00:06.342464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:06.339997 5036 scope.go:122] "RemoveContainer" containerID="f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51" Nov 01 22:00:06.342464 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:06.340550 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd_odh-ai-gateway-infra(ae5f03cc-4d69-4383-b16f-abb324a0f59a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" Nov 01 22:00:06.915734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:06.915687 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:00:06.997274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:06.997228 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.067541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.066681 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:00:07.068347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.068310 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.079562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.079525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.131465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.123838 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:00:07.168523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.167845 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.204470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.203544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.470750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.466612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:07.786857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.786292 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:00:07.903181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.903135 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.925231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.921430 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.925231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:07.921920 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:00:08.028609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.028546 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.029566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.029536 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:00:08.055535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.048940 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 22:00:08.271036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.270898 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.278710 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.281543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.281876 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.282139 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.313892 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:00:08.318525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.313957 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:00:08.337468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.321045 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.367492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.341541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.367492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.341912 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:00:08.375483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.367778 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:00:08.375483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.367886 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:00:08.375483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.367924 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:00:08.566353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.539685 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 01 22:00:08.625350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.625258 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:00:08.629577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.629515 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.825498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.813415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:00:08.919568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.896300 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:00:08.919568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.918879 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.920043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.920019 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:00:08.921679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:08.921654 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.035377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.035291 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.293976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.293933 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.295153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.295116 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:00:09.295966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.295927 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:00:09.297861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.297828 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:00:09.321352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.320091 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:00:09.367513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.352622 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:00:09.367951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.367843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:00:09.369356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.369312 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:00:09.849602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.849550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.931328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.931284 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.932534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.932503 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.971510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:09.971481 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:00:10.157527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.090728 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:00:10.323467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.261003 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:00:10.323467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.298600 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:00:10.323467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.298558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:00:10.323467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.300134 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.323467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.300371 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.373157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.373119 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:00:10.397471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.381129 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:00:10.406497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.405685 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:10.796436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.796026 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:10.899166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.899120 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.952602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:10.952556 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.164592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.164484 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.255751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.255701 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.323897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.323854 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:11.359089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.325548 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:00:11.359527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.359487 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:00:11.361599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.361567 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.416717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.416596 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:00:11.444333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.444288 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:00:11.476814 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:11.447614 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.016s" Nov 01 22:00:11.477131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.477112 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:00:11.477281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.477270 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:00:11.719729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.719598 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.791474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.778401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.943471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.906618 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:00:11.943471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.941293 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.961028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.961001 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:00:11.984482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:11.984373 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.289898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:12.289850 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.360059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:12.359635 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.363310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:12.362502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.427380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:12.425550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.859466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:12.855774 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:00:13.041468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.033037 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:13.066614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.066561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.081679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.081590 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.142635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.142524 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:00:13.364093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.360520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.364093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.361636 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:00:13.390468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.376548 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.501691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.466144 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:00:13.501691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.466221 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.624640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.624579 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.754737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.751705 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.926468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:13.926147 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.052567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.052523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:00:14.083220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.083179 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.185089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.173728 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:00:14.187102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.173763 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 01 22:00:14.187102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.173799 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.200463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.187385 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:00:14.225620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.225571 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:00:14.398527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.397760 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.398527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.398317 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:00:14.400547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.400501 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.417939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.416038 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:00:14.420091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.420055 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:00:14.449669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.416069 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 22:00:14.665937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.665844 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.752683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.752016 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:00:14.817164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.810742 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Nov 01 22:00:14.858465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.829389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:00:14.858465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.841515 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:14.936518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.934952 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:14.936518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:14.935716 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:00:15.096470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.075061 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:15.266547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.266470 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:00:15.290203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.290175 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:00:15.290498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.290483 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:00:15.304846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.304808 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:15.312876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.312839 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:15.314090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.314062 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:00:15.348722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.348692 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:00:15.419660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.414766 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:00:15.574540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.574507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:15.629552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.608816 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:00:15.629552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.608890 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:00:15.678709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.678649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:15.742343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.742308 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:00:15.743489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.743438 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:00:15.789528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.789482 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:15.822079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.821935 5036 scope.go:122] "RemoveContainer" containerID="9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2" Nov 01 22:00:15.822862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.822822 5036 scope.go:122] "RemoveContainer" containerID="1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce" Nov 01 22:00:15.934797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.933032 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:00:15.999745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:15.999707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:16.195108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.194334 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:00:16.253803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.252856 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:00:16.262999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.262970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:16.265284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.265240 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:16.334118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.334086 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr"] Nov 01 22:00:16.361854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.354128 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1aee4437-nlzsr"] Nov 01 22:00:16.448113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.448032 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" path="/var/lib/kubelet/pods/a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6/volumes" Nov 01 22:00:16.448430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.448410 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 22:00:16.448986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.448781 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" podUID="2eab6863-80f1-49e0-99b1-ac5a6b601000" containerName="istio-proxy" containerID="cri-o://eaffbe29ecc78aab10fcc8be79b01219cb9a2d0008d4f41210ba23af442f0dab" gracePeriod=30 Nov 01 22:00:16.448986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:16.448909 5036 scope.go:122] "RemoveContainer" containerID="297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1" Nov 01 22:00:17.045047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.045013 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:17.045942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.045916 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:17.220924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.220882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:17.255464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.229749 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\") pod \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " Nov 01 22:00:17.255464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.245805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\" (UniqueName: \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\") pod \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\" (UID: \"ae5f03cc-4d69-4383-b16f-abb324a0f59a\") " Nov 01 22:00:17.255464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.235482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca" pod "ae5f03cc-4d69-4383-b16f-abb324a0f59a" (UID: "ae5f03cc-4d69-4383-b16f-abb324a0f59a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:17.266481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.255964 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg" pod "ae5f03cc-4d69-4383-b16f-abb324a0f59a" (UID: "ae5f03cc-4d69-4383-b16f-abb324a0f59a"). InnerVolumeSpecName "kube-api-access-6vwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:17.352481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.348776 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwxg\" (UniqueName: \"kubernetes.io/projected/ae5f03cc-4d69-4383-b16f-abb324a0f59a-kube-api-access-6vwxg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:17.352481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.348820 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5f03cc-4d69-4383-b16f-abb324a0f59a-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:17.386515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.385856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:17.436479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.432420 5036 scope.go:122] "RemoveContainer" containerID="6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22" Nov 01 22:00:17.538342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:17.527998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:18.016305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.016274 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerID="3de775e4761b1d9395b2414075fceb1ee3529e38b8b13c11f97b0f4c5e1dbbfd" exitCode=60 Nov 01 22:00:18.016525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.016335 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerDied","Data":"3de775e4761b1d9395b2414075fceb1ee3529e38b8b13c11f97b0f4c5e1dbbfd"} Nov 01 22:00:18.016525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.016403 5036 scope.go:122] "RemoveContainer" containerID="297fe7c32847f370884fbffce3d6fa6012d1cc7110c146b9c7d7136c4c501bc1" Nov 01 22:00:18.016941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.016925 5036 scope.go:122] "RemoveContainer" containerID="3de775e4761b1d9395b2414075fceb1ee3529e38b8b13c11f97b0f4c5e1dbbfd" Nov 01 22:00:18.017295 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:18.017277 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x_odh-ai-gateway-infra(6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" Nov 01 22:00:18.020868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.020842 5036 generic.go:356] "Generic (PLEG): container finished" podID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerID="2043f661dbf68a2da49534749c94c6b21bc8f47b660915a8fd09a9ac0c9d6722" exitCode=60 Nov 01 22:00:18.020959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.020874 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerDied","Data":"2043f661dbf68a2da49534749c94c6b21bc8f47b660915a8fd09a9ac0c9d6722"} Nov 01 22:00:18.021373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.021353 5036 scope.go:122] "RemoveContainer" containerID="2043f661dbf68a2da49534749c94c6b21bc8f47b660915a8fd09a9ac0c9d6722" Nov 01 22:00:18.021812 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:18.021784 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c_odh-ai-gateway-infra(739c36a8-4f72-4837-93a0-b2d3f0402568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" Nov 01 22:00:18.023322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.023303 5036 generic.go:356] "Generic (PLEG): container finished" podID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerID="91f11b3ef2848f38409ad06dd35398569ff871863614a282ca64ad46675fdde2" exitCode=60 Nov 01 22:00:18.023410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.023379 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerDied","Data":"91f11b3ef2848f38409ad06dd35398569ff871863614a282ca64ad46675fdde2"} Nov 01 22:00:18.023885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.023855 5036 scope.go:122] "RemoveContainer" containerID="91f11b3ef2848f38409ad06dd35398569ff871863614a282ca64ad46675fdde2" Nov 01 22:00:18.024256 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:18.024227 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l_odh-ai-gateway-infra(aed6254e-e206-408f-91a3-36ad5b4fbf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" Nov 01 22:00:18.025117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.025098 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" event={"ID":"ae5f03cc-4d69-4383-b16f-abb324a0f59a","Type":"ContainerDied","Data":"78772ed0c1599666f5240bfaa91b13f5e88bfded0b42b0336a280123939886c7"} Nov 01 22:00:18.025202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.025175 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd" Nov 01 22:00:18.036755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.036731 5036 scope.go:122] "RemoveContainer" containerID="9c932a24027b6041c871868b452532f954d16c2bf2a04e427fd1a9d6fea559c2" Nov 01 22:00:18.054259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.054234 5036 scope.go:122] "RemoveContainer" containerID="1891ec0109e27ba308de8184da219cd83e27f32f2510d442937e6cf7336a12ce" Nov 01 22:00:18.072597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.072579 5036 scope.go:122] "RemoveContainer" containerID="f02e90a40250948917c05ddd2cee982a4572ca109e588ae385535862106a8d51" Nov 01 22:00:18.099601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.099582 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:18.112317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.112294 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-1aee4437-29892840-jnnwd"] Nov 01 22:00:18.330976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.330943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:18.443840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:18.443762 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" path="/var/lib/kubelet/pods/ae5f03cc-4d69-4383-b16f-abb324a0f59a/volumes" Nov 01 22:00:18.858807 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:18.858772 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:00:19.047097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.047060 5036 generic.go:356] "Generic (PLEG): container finished" podID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerID="2c6ae421f265e94777c587063053a5c55a2e3c5192ddfcbc04e305bf60af8a8b" exitCode=60 Nov 01 22:00:19.047278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.047134 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerDied","Data":"2c6ae421f265e94777c587063053a5c55a2e3c5192ddfcbc04e305bf60af8a8b"} Nov 01 22:00:19.047278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.047198 5036 scope.go:122] "RemoveContainer" containerID="6bedfce7fff19af6c411821021a7efd9f2dd57580021c3ea5c0c803d40fa7e22" Nov 01 22:00:19.047721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.047700 5036 scope.go:122] "RemoveContainer" containerID="2c6ae421f265e94777c587063053a5c55a2e3c5192ddfcbc04e305bf60af8a8b" Nov 01 22:00:19.048166 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:19.048137 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-s7996_odh-ai-gateway-infra(7897dce5-113f-4f80-a61a-aa25e14bf232)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" Nov 01 22:00:19.049813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.049382 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:19.071597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.071563 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:19.132408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.132370 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:19.188614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.188568 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn"] Nov 01 22:00:19.192825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.190569 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" containerName="revoke-keys" Nov 01 22:00:19.195691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.195659 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" containerName="revoke-keys" Nov 01 22:00:19.195959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.195944 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.196057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196045 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.196150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196140 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.196226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196216 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.196746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196725 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24c62fc-0e0b-45a8-a64a-e2bdedeb63a6" containerName="revoke-keys" Nov 01 22:00:19.196864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196852 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.196948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.196939 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5f03cc-4d69-4383-b16f-abb324a0f59a" containerName="cleanup" Nov 01 22:00:19.226719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.221688 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn"] Nov 01 22:00:19.226719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.221849 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.230711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.229804 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:00:19.297542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.296456 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.297542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.296709 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.297542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.296899 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrnwt\" (UniqueName: \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.316420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.308936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:19.389231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.388535 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:19.400480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.398684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrnwt\" (UniqueName: \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.400480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.398805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.400480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.399066 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.400480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.400198 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.405498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.403377 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.449095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.439303 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrnwt\" (UniqueName: \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\") pod \"maas-api-revoke-keys-e2e-derive-d5713372-ws4tn\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.474387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.474350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:19.503554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.499993 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\" (UniqueName: \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\") pod \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " Nov 01 22:00:19.503554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.500106 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\") pod \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\" (UID: \"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4\") " Nov 01 22:00:19.503554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.502091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca" pod "6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" (UID: "6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:19.506514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.503986 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt" pod "6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" (UID: "6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4"). InnerVolumeSpecName "kube-api-access-cv2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:19.546115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.545711 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:19.550255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.550189 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.604574 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\" (UniqueName: \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\") pod \"739c36a8-4f72-4837-93a0-b2d3f0402568\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.604666 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\") pod \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.604797 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\") pod \"739c36a8-4f72-4837-93a0-b2d3f0402568\" (UID: \"739c36a8-4f72-4837-93a0-b2d3f0402568\") " Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.604859 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\" (UniqueName: \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\") pod \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\" (UID: \"aed6254e-e206-408f-91a3-36ad5b4fbf5c\") " Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.605650 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2wt\" (UniqueName: \"kubernetes.io/projected/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-kube-api-access-cv2wt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.606058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.605673 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.611640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.607033 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca" pod "aed6254e-e206-408f-91a3-36ad5b4fbf5c" (UID: "aed6254e-e206-408f-91a3-36ad5b4fbf5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:19.611951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.611841 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca" pod "739c36a8-4f72-4837-93a0-b2d3f0402568" (UID: "739c36a8-4f72-4837-93a0-b2d3f0402568"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:19.612086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.612053 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:19.613371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.613333 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5" pod "aed6254e-e206-408f-91a3-36ad5b4fbf5c" (UID: "aed6254e-e206-408f-91a3-36ad5b4fbf5c"). InnerVolumeSpecName "kube-api-access-8c4k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:19.615961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.615929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q" pod "739c36a8-4f72-4837-93a0-b2d3f0402568" (UID: "739c36a8-4f72-4837-93a0-b2d3f0402568"). InnerVolumeSpecName "kube-api-access-tws6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:19.707472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.707232 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4k5\" (UniqueName: \"kubernetes.io/projected/aed6254e-e206-408f-91a3-36ad5b4fbf5c-kube-api-access-8c4k5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.707472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.707268 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tws6q\" (UniqueName: \"kubernetes.io/projected/739c36a8-4f72-4837-93a0-b2d3f0402568-kube-api-access-tws6q\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.707472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.707287 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed6254e-e206-408f-91a3-36ad5b4fbf5c-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.707472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.707301 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/739c36a8-4f72-4837-93a0-b2d3f0402568-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.929126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:19.929085 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn"] Nov 01 22:00:19.929597 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:00:19.929559 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11625c83_73c9_4e10_b08e_ca3131aa9084.slice/crio-0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005 WatchSource:0}: Error finding container 0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005: Status 404 returned error can't find the container with id 0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005 Nov 01 22:00:20.062159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.062125 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" Nov 01 22:00:20.062330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.062151 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c" event={"ID":"739c36a8-4f72-4837-93a0-b2d3f0402568","Type":"ContainerDied","Data":"92c5ece31a42e1c6b2505c45ca4a3004eddb9ae2dcfcaf2c69e42c03ed4813ba"} Nov 01 22:00:20.062330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.062227 5036 scope.go:122] "RemoveContainer" containerID="2043f661dbf68a2da49534749c94c6b21bc8f47b660915a8fd09a9ac0c9d6722" Nov 01 22:00:20.065779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.065746 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" Nov 01 22:00:20.065933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.065804 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l" event={"ID":"aed6254e-e206-408f-91a3-36ad5b4fbf5c","Type":"ContainerDied","Data":"72f2a3e5dea49a5748598cc4426c82d5a07d019cb89637b3f05adc8fbab63c2c"} Nov 01 22:00:20.075017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.074956 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" event={"ID":"11625c83-73c9-4e10-b08e-ca3131aa9084","Type":"ContainerStarted","Data":"0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005"} Nov 01 22:00:20.082827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.082805 5036 scope.go:122] "RemoveContainer" containerID="91f11b3ef2848f38409ad06dd35398569ff871863614a282ca64ad46675fdde2" Nov 01 22:00:20.085630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.084609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" event={"ID":"6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4","Type":"ContainerDied","Data":"e770e0e600a14e3246db5bb1f3f5e82703b4a0a332ea368e4bfedcf5306ac69b"} Nov 01 22:00:20.085630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.084713 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x" Nov 01 22:00:20.091982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.090517 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:20.108815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.108792 5036 scope.go:122] "RemoveContainer" containerID="3de775e4761b1d9395b2414075fceb1ee3529e38b8b13c11f97b0f4c5e1dbbfd" Nov 01 22:00:20.149906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.149858 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:20.157154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.157097 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892840-kfs8x"] Nov 01 22:00:20.163918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.163888 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:20.174334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.173355 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892840-29f5c"] Nov 01 22:00:20.185492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.185468 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:20.196107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.196080 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892840-5sc8l"] Nov 01 22:00:20.229501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.229465 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:20.356560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.356236 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:20.435812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.435718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\") pod \"7897dce5-113f-4f80-a61a-aa25e14bf232\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " Nov 01 22:00:20.436280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.435898 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\" (UniqueName: \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\") pod \"7897dce5-113f-4f80-a61a-aa25e14bf232\" (UID: \"7897dce5-113f-4f80-a61a-aa25e14bf232\") " Nov 01 22:00:20.436280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.436159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca" pod "7897dce5-113f-4f80-a61a-aa25e14bf232" (UID: "7897dce5-113f-4f80-a61a-aa25e14bf232"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:20.436632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.436599 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7897dce5-113f-4f80-a61a-aa25e14bf232-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:20.438338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.438314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn" pod "7897dce5-113f-4f80-a61a-aa25e14bf232" (UID: "7897dce5-113f-4f80-a61a-aa25e14bf232"). InnerVolumeSpecName "kube-api-access-xbrwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:20.444656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.444634 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" path="/var/lib/kubelet/pods/6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4/volumes" Nov 01 22:00:20.445060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.445047 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" path="/var/lib/kubelet/pods/739c36a8-4f72-4837-93a0-b2d3f0402568/volumes" Nov 01 22:00:20.445439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.445427 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" path="/var/lib/kubelet/pods/aed6254e-e206-408f-91a3-36ad5b4fbf5c/volumes" Nov 01 22:00:20.538160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.538123 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrwn\" (UniqueName: \"kubernetes.io/projected/7897dce5-113f-4f80-a61a-aa25e14bf232-kube-api-access-xbrwn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:20.941944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:20.941912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:21.103795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.103758 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" event={"ID":"7897dce5-113f-4f80-a61a-aa25e14bf232","Type":"ContainerDied","Data":"a4938a631e3e24dd3be65251156c574dc593b1ac4d889cab9fe5d598f9c012d9"} Nov 01 22:00:21.104134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.103798 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996" Nov 01 22:00:21.104134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.104123 5036 scope.go:122] "RemoveContainer" containerID="2c6ae421f265e94777c587063053a5c55a2e3c5192ddfcbc04e305bf60af8a8b" Nov 01 22:00:21.110069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.110040 5036 generic.go:356] "Generic (PLEG): container finished" podID="11625c83-73c9-4e10-b08e-ca3131aa9084" containerID="b5a9aafbd3bed65aa6f5b50364d61da6af54fac551f4cf19f480a569e3d012e1" exitCode=0 Nov 01 22:00:21.110205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.110178 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" event={"ID":"11625c83-73c9-4e10-b08e-ca3131aa9084","Type":"ContainerDied","Data":"b5a9aafbd3bed65aa6f5b50364d61da6af54fac551f4cf19f480a569e3d012e1"} Nov 01 22:00:21.133504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.133469 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:21.139864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.139840 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-s7996"] Nov 01 22:00:21.723687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:21.723647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:22.132756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.132718 5036 generic.go:356] "Generic (PLEG): container finished" podID="2eab6863-80f1-49e0-99b1-ac5a6b601000" containerID="eaffbe29ecc78aab10fcc8be79b01219cb9a2d0008d4f41210ba23af442f0dab" exitCode=0 Nov 01 22:00:22.133094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.132794 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" event={"ID":"2eab6863-80f1-49e0-99b1-ac5a6b601000","Type":"ContainerDied","Data":"eaffbe29ecc78aab10fcc8be79b01219cb9a2d0008d4f41210ba23af442f0dab"} Nov 01 22:00:22.186488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.182973 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 22:00:22.265745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.265701 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.265940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.265804 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.265940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.265867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.266081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.265937 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.266081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.266064 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.266383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.266330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:00:22.266563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.266424 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:00:22.266563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.266534 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:22.266906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.266885 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.267191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.267171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:00:22.267295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.267278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.267783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.267761 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.267889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.267874 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\") pod \"2eab6863-80f1-49e0-99b1-ac5a6b601000\" (UID: \"2eab6863-80f1-49e0-99b1-ac5a6b601000\") " Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.268919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.268990 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269371 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269392 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269406 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab6863-80f1-49e0-99b1-ac5a6b601000-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269420 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269433 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.269482 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.270999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.270594 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:00:22.275620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.275592 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "kube-api-access-zwh7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:22.278707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.278624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token" pod "2eab6863-80f1-49e0-99b1-ac5a6b601000" (UID: "2eab6863-80f1-49e0-99b1-ac5a6b601000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:22.347824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.347798 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:22.370868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.370833 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.371058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.370873 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwh7l\" (UniqueName: \"kubernetes.io/projected/2eab6863-80f1-49e0-99b1-ac5a6b601000-kube-api-access-zwh7l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.371058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.370891 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab6863-80f1-49e0-99b1-ac5a6b601000-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.448515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.448429 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" path="/var/lib/kubelet/pods/7897dce5-113f-4f80-a61a-aa25e14bf232/volumes" Nov 01 22:00:22.472075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.472048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\") pod \"11625c83-73c9-4e10-b08e-ca3131aa9084\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " Nov 01 22:00:22.472214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.472138 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\") pod \"11625c83-73c9-4e10-b08e-ca3131aa9084\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " Nov 01 22:00:22.472329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.472314 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\" (UniqueName: \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\") pod \"11625c83-73c9-4e10-b08e-ca3131aa9084\" (UID: \"11625c83-73c9-4e10-b08e-ca3131aa9084\") " Nov 01 22:00:22.472488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.472464 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca" pod "11625c83-73c9-4e10-b08e-ca3131aa9084" (UID: "11625c83-73c9-4e10-b08e-ca3131aa9084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:22.472892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.472871 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.474422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.474390 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca" pod "11625c83-73c9-4e10-b08e-ca3131aa9084" (UID: "11625c83-73c9-4e10-b08e-ca3131aa9084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:00:22.474577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.474547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt" pod "11625c83-73c9-4e10-b08e-ca3131aa9084" (UID: "11625c83-73c9-4e10-b08e-ca3131aa9084"). InnerVolumeSpecName "kube-api-access-lrnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:22.574058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.574028 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrnwt\" (UniqueName: \"kubernetes.io/projected/11625c83-73c9-4e10-b08e-ca3131aa9084-kube-api-access-lrnwt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.574058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.574056 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11625c83-73c9-4e10-b08e-ca3131aa9084-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:22.642328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:22.642291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:23.144610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.144572 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" event={"ID":"11625c83-73c9-4e10-b08e-ca3131aa9084","Type":"ContainerDied","Data":"0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005"} Nov 01 22:00:23.145083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.144628 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0364cb802d8c4bd37896f0fe4a9df080a4a3fc3a6120a528aff9c30c455c9005" Nov 01 22:00:23.145083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.144588 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn" Nov 01 22:00:23.146377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.146349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" event={"ID":"2eab6863-80f1-49e0-99b1-ac5a6b601000","Type":"ContainerDied","Data":"a8035599dda09768c2753c772193ca7f206f2ff5bb453a074b579171e4852980"} Nov 01 22:00:23.146569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.146404 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg" Nov 01 22:00:23.146569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.146405 5036 scope.go:122] "RemoveContainer" containerID="eaffbe29ecc78aab10fcc8be79b01219cb9a2d0008d4f41210ba23af442f0dab" Nov 01 22:00:23.171574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.171551 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 22:00:23.181708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.181678 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1aee4437-gw-openshift-default-6795596c97-zz4dg"] Nov 01 22:00:23.434700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:23.434660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:24.297467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:24.296996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:24.437795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:24.437762 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:24.440689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:24.440666 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eab6863-80f1-49e0-99b1-ac5a6b601000" path="/var/lib/kubelet/pods/2eab6863-80f1-49e0-99b1-ac5a6b601000/volumes" Nov 01 22:00:24.807546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:24.807513 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:25.233376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:25.233280 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:25.376491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:25.369718 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:26.065321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:26.065278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:26.928010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:26.927973 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:27.827070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:27.827035 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:28.635045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:28.634993 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:29.217519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:29.217417 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:29.356847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:29.356797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:29.483766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:29.483673 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:30.294979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:30.294937 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:30.574513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:30.574390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:30.707418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:30.707381 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:31.352346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:31.352313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:32.017200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.017159 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:32.159108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.159041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:32.196705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.196666 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn"] Nov 01 22:00:32.202923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.202896 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5713372-ws4tn"] Nov 01 22:00:32.303971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.303937 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:32.504927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.504868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:32.566717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:32.566623 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11625c83-73c9-4e10-b08e-ca3131aa9084" path="/var/lib/kubelet/pods/11625c83-73c9-4e10-b08e-ca3131aa9084/volumes" Nov 01 22:00:33.153043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:33.153007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:33.552510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:33.552472 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:33.729381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:33.729344 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:33.854777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:33.854687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:34.440469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:34.438064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:35.112121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:35.112078 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:35.776085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:35.776045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:35.901398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:35.901358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:36.027128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:36.027027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:37.022646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:37.022613 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:37.195981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:37.195941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:37.322424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:37.322328 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:37.858778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:37.858728 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:38.394542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:38.394503 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:38.520827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:38.520792 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:38.887297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:38.887249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:39.698852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:39.698808 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:40.555223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:40.555190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:41.250869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:41.250832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:41.947911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:41.947873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:42.043633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:42.043595 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2"] Nov 01 22:00:42.051676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:42.051648 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90cfc7a4-2wzp2"] Nov 01 22:00:42.440161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:42.440124 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bc59da-6939-4dd7-9a28-fd8a44ee92fa" path="/var/lib/kubelet/pods/f8bc59da-6939-4dd7-9a28-fd8a44ee92fa/volumes" Nov 01 22:00:42.646235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:42.646199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:43.338415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:43.338378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:44.024890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:44.024850 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:44.152488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:44.152419 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:44.277097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:44.276999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:44.496817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:44.496781 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:00:45.049332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:45.049293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:45.882094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:45.882052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:47.128040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:47.127986 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p"] Nov 01 22:00:47.142209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:47.142113 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:47.150558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:47.150539 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339d468-fj97p"] Nov 01 22:00:48.057785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:48.057744 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8"] Nov 01 22:00:48.069202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:48.069167 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54775354-4j6l8"] Nov 01 22:00:48.355594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:48.355389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:48.562468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:48.536352 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8d920b-088b-4180-9d1b-91741afa7c7a" path="/var/lib/kubelet/pods/6d8d920b-088b-4180-9d1b-91741afa7c7a/volumes" Nov 01 22:00:48.563852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:48.563821 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9384ed2e-f8bf-4f7f-9ae2-dd868e325237" path="/var/lib/kubelet/pods/9384ed2e-f8bf-4f7f-9ae2-dd868e325237/volumes" Nov 01 22:00:48.923732 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:00:48.923678 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:00:49.559347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:49.559307 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:50.031917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:50.031887 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb"] Nov 01 22:00:50.037762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:50.037734 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f245760-vfmnb"] Nov 01 22:00:50.439904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:50.439829 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b463a9a-b46a-4612-9679-1d56248020d6" path="/var/lib/kubelet/pods/3b463a9a-b46a-4612-9679-1d56248020d6/volumes" Nov 01 22:00:50.462497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:50.462404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:51.248878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:51.248837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:52.030015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.029986 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx"] Nov 01 22:00:52.032312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.032277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:52.036368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.036345 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j"] Nov 01 22:00:52.052388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.052354 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e134d0c-jv2tx"] Nov 01 22:00:52.060013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.059979 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc71139-vvq4j"] Nov 01 22:00:52.440502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.440397 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31" path="/var/lib/kubelet/pods/ec8720da-5c65-4ddc-ac8a-0a3ab6f74d31/volumes" Nov 01 22:00:52.440907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.440891 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5361da4-de94-4a09-80d5-f179db6548f0" path="/var/lib/kubelet/pods/f5361da4-de94-4a09-80d5-f179db6548f0/volumes" Nov 01 22:00:52.701782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:52.701680 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:53.813359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:53.813308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:54.309227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:54.309188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:54.740401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:54.740298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:54.872907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:54.872869 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:55.113922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.113883 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:00:55.114952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114925 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2eab6863-80f1-49e0-99b1-ac5a6b601000" containerName="istio-proxy" Nov 01 22:00:55.114952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114951 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eab6863-80f1-49e0-99b1-ac5a6b601000" containerName="istio-proxy" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114968 5036 cpu_manager.go:414] "containerMap: removing container" podUID="11625c83-73c9-4e10-b08e-ca3131aa9084" containerName="revoke-keys" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114977 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11625c83-73c9-4e10-b08e-ca3131aa9084" containerName="revoke-keys" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114991 5036 cpu_manager.go:414] "containerMap: removing container" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.114999 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115034 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115041 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115052 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115059 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115070 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115077 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115087 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115094 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115111 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115118 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115137 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115144 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115162 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115169 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115182 5036 cpu_manager.go:414] "containerMap: removing container" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115188 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115210 5036 cpu_manager.go:414] "containerMap: removing container" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115217 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115374 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115389 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115400 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115410 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:55.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.115421 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116019 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eab6863-80f1-49e0-99b1-ac5a6b601000" containerName="istio-proxy" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116038 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="11625c83-73c9-4e10-b08e-ca3131aa9084" containerName="revoke-keys" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116089 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116101 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116113 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:55.116571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.116124 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:55.122769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.122736 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.134603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.134576 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:00:55.152694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.152672 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-e2bf63-openshift-default-dockercfg-tjbqw\"" Nov 01 22:00:55.189898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.189863 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:55.236466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889qj\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.236666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236527 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.236736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236717 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.236800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.236869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.244544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236942 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.244544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.236988 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.244544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.237106 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.244544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.237172 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.325465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.320964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339138 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339284 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339346 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339372 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889qj\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.339609 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.340077 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.341404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.341067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.342815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.342791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.343237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.343216 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.369035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.359766 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.388015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.377791 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.388015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.387941 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.392202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.392133 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889qj\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.403782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.403713 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.460631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.460557 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:55.883922 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:00:55.883881 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c5b19f_c305_43fb_bd5b_43f6bcffbd8c.slice/crio-ae3a70901340601e0d83986596c10626eb90127913f52dfb4e5be8bf96705689 WatchSource:0}: Error finding container ae3a70901340601e0d83986596c10626eb90127913f52dfb4e5be8bf96705689: Status 404 returned error can't find the container with id ae3a70901340601e0d83986596c10626eb90127913f52dfb4e5be8bf96705689 Nov 01 22:00:55.885532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.885503 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:00:55.887067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.887033 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:00:55.887158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.887108 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:00:55.887158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:55.887152 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:00:56.032734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:56.032702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:56.752392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:56.752354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerStarted","Data":"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99"} Nov 01 22:00:56.752638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:56.752422 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerStarted","Data":"ae3a70901340601e0d83986596c10626eb90127913f52dfb4e5be8bf96705689"} Nov 01 22:00:56.784205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:56.784144 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podStartSLOduration=1.7841302639999999 podStartE2EDuration="1.784130264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:56.776692622 +0000 UTC m=+1010.469673110" watchObservedRunningTime="2026-11-01 22:00:56.784130264 +0000 UTC m=+1010.477110750" Nov 01 22:00:56.829970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:56.829933 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:57.174424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.174384 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:57.302469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.300343 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:57.462501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.462423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:57.468633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.468608 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:57.556945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.556912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:57.726144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.726048 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:57.762496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.762469 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:57.763425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:57.763407 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:00:58.037991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.037959 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq"] Nov 01 22:00:58.048128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.047795 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9158db-hhcsq"] Nov 01 22:00:58.145780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.145740 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:58.451126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.451025 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b6ad9e-6692-4133-bbcb-88dfadad1578" path="/var/lib/kubelet/pods/84b6ad9e-6692-4133-bbcb-88dfadad1578/volumes" Nov 01 22:00:58.532162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.532128 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:58.784599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:58.784542 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:59.026662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.026626 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:59.167089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.167000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:59.178158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178126 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx"] Nov 01 22:00:59.178855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178839 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:59.178933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178857 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7897dce5-113f-4f80-a61a-aa25e14bf232" containerName="cleanup" Nov 01 22:00:59.178933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178884 5036 cpu_manager.go:414] "containerMap: removing container" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:59.178933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178889 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:59.179040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.178996 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed6254e-e206-408f-91a3-36ad5b4fbf5c" containerName="cleanup" Nov 01 22:00:59.179040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.179009 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83ca7d-ea7f-4461-abc6-326f4ca0b7d4" containerName="cleanup" Nov 01 22:00:59.179040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.179016 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="739c36a8-4f72-4837-93a0-b2d3f0402568" containerName="cleanup" Nov 01 22:00:59.184319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.184295 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.188266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.188243 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:00:59.189541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.189520 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:00:59.189654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.189612 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:00:59.189654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.189521 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:00:59.191301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.191273 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx"] Nov 01 22:00:59.345275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.345239 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.345452 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.345332 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chg86\" (UniqueName: \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.345452 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.345424 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.447434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.447343 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.447434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.447413 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chg86\" (UniqueName: \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.447681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.447493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.448166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.448141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.449914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.449893 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.469872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.469843 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chg86\" (UniqueName: \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\") pod \"maas-api-revoke-sub-48ee0976-4q6sx\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.517875 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:00:59.739299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.739202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:59.851583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:00:59.851544 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx"] Nov 01 22:01:00.045675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.043782 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc"] Nov 01 22:01:00.053560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.053529 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d8fb5e-h8mkc"] Nov 01 22:01:00.312144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.295582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:00.430887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.430854 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:00.449103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.448342 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c00f306-c89e-42b2-8a67-82bdffcae3a7" path="/var/lib/kubelet/pods/9c00f306-c89e-42b2-8a67-82bdffcae3a7/volumes" Nov 01 22:01:00.811926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.811891 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" event={"ID":"f84651b4-5a93-4c47-8677-034aed60f775","Type":"ContainerStarted","Data":"cc68c0bf9bc8ea6f0eebc5f3bc90cf6ab74c911d12d0ac2ac77c37d51baa263e"} Nov 01 22:01:00.812493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.811949 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" event={"ID":"f84651b4-5a93-4c47-8677-034aed60f775","Type":"ContainerStarted","Data":"10c7b8f5c78398405fa6ff4392eff09c2ab1b6a1a78a4b29c0487e804e766c48"} Nov 01 22:01:00.836676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:00.836593 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" podStartSLOduration=1.836580471 podStartE2EDuration="1.836580471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:00.831123605 +0000 UTC m=+1014.524104091" watchObservedRunningTime="2026-11-01 22:01:00.836580471 +0000 UTC m=+1014.529560957" Nov 01 22:01:01.062242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:01.062188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:01.828289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:01.828251 5036 generic.go:356] "Generic (PLEG): container finished" podID="f84651b4-5a93-4c47-8677-034aed60f775" containerID="cc68c0bf9bc8ea6f0eebc5f3bc90cf6ab74c911d12d0ac2ac77c37d51baa263e" exitCode=0 Nov 01 22:01:01.828704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:01.828346 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" event={"ID":"f84651b4-5a93-4c47-8677-034aed60f775","Type":"ContainerDied","Data":"cc68c0bf9bc8ea6f0eebc5f3bc90cf6ab74c911d12d0ac2ac77c37d51baa263e"} Nov 01 22:01:02.119758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:02.119461 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:02.256835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:02.256796 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:02.385522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:02.384979 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:03.021904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.021879 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:01:03.028069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.028045 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf"] Nov 01 22:01:03.034700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.034678 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd6c32b6-scddf"] Nov 01 22:01:03.051174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.051143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:03.069833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.069800 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\") pod \"f84651b4-5a93-4c47-8677-034aed60f775\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " Nov 01 22:01:03.070113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.070095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\" (UniqueName: \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\") pod \"f84651b4-5a93-4c47-8677-034aed60f775\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " Nov 01 22:01:03.070217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.070186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca" pod "f84651b4-5a93-4c47-8677-034aed60f775" (UID: "f84651b4-5a93-4c47-8677-034aed60f775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:01:03.072181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.072155 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\") pod \"f84651b4-5a93-4c47-8677-034aed60f775\" (UID: \"f84651b4-5a93-4c47-8677-034aed60f775\") " Nov 01 22:01:03.073323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.073249 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86" pod "f84651b4-5a93-4c47-8677-034aed60f775" (UID: "f84651b4-5a93-4c47-8677-034aed60f775"). InnerVolumeSpecName "kube-api-access-chg86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:03.073818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.073796 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:03.099097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.093863 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca" pod "f84651b4-5a93-4c47-8677-034aed60f775" (UID: "f84651b4-5a93-4c47-8677-034aed60f775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:01:03.176194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.176158 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chg86\" (UniqueName: \"kubernetes.io/projected/f84651b4-5a93-4c47-8677-034aed60f775-kube-api-access-chg86\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:03.176194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.176194 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84651b4-5a93-4c47-8677-034aed60f775-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:03.576921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.576882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:03.639684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.639643 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:01:03.640356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.640342 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f84651b4-5a93-4c47-8677-034aed60f775" containerName="revoke-keys" Nov 01 22:01:03.640401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.640358 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84651b4-5a93-4c47-8677-034aed60f775" containerName="revoke-keys" Nov 01 22:01:03.640508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.640498 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84651b4-5a93-4c47-8677-034aed60f775" containerName="revoke-keys" Nov 01 22:01:03.644999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.644977 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.650326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.650298 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-2c5135-dockercfg-t8gkr\"" Nov 01 22:01:03.650558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.650539 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-2c5135\"" Nov 01 22:01:03.655775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.655749 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:01:03.711982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.711948 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:03.808467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.806842 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6dc\" (UniqueName: \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.808467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.807111 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.917553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.915922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.917553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.916117 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6dc\" (UniqueName: \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.917553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.917500 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.949632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.946121 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" event={"ID":"f84651b4-5a93-4c47-8677-034aed60f775","Type":"ContainerDied","Data":"10c7b8f5c78398405fa6ff4392eff09c2ab1b6a1a78a4b29c0487e804e766c48"} Nov 01 22:01:03.949632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.946214 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c7b8f5c78398405fa6ff4392eff09c2ab1b6a1a78a4b29c0487e804e766c48" Nov 01 22:01:03.949632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.946272 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx" Nov 01 22:01:03.971463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.961607 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6dc\" (UniqueName: \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\") pod \"payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:03.971463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:03.971179 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:04.068360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.068303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:04.331287 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:01:04.331254 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01302463_9a3a_46cf_a162_d83e9a846249.slice/crio-eabc2de46bd40305a4d0ffd459ab32c160ba82f44af2b0d742650d3893e6a718 WatchSource:0}: Error finding container eabc2de46bd40305a4d0ffd459ab32c160ba82f44af2b0d742650d3893e6a718: Status 404 returned error can't find the container with id eabc2de46bd40305a4d0ffd459ab32c160ba82f44af2b0d742650d3893e6a718 Nov 01 22:01:04.336465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.336421 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:01:04.441398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.441370 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c" path="/var/lib/kubelet/pods/d9dcbe9e-ef62-45fd-9ac3-e622a5059e6c/volumes" Nov 01 22:01:04.970794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.969647 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" event={"ID":"01302463-9a3a-46cf-a162-d83e9a846249","Type":"ContainerStarted","Data":"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c"} Nov 01 22:01:04.971015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.970825 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:04.971015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:04.970845 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" event={"ID":"01302463-9a3a-46cf-a162-d83e9a846249","Type":"ContainerStarted","Data":"eabc2de46bd40305a4d0ffd459ab32c160ba82f44af2b0d742650d3893e6a718"} Nov 01 22:01:05.015615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:05.015564 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" podStartSLOduration=2.015550003 podStartE2EDuration="2.015550003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:05.010635097 +0000 UTC m=+1018.703615585" watchObservedRunningTime="2026-11-01 22:01:05.015550003 +0000 UTC m=+1018.708530489" Nov 01 22:01:05.026268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:05.026234 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:05.036626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:05.036594 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4"] Nov 01 22:01:05.043857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:05.043827 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1b603-8rws4"] Nov 01 22:01:05.920494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:05.919226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:06.032335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:06.032300 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp"] Nov 01 22:01:06.038075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:06.038051 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876f6fe-wb5hp"] Nov 01 22:01:06.441940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:06.441911 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66478636-93a1-4486-b7bd-65b95ef1d150" path="/var/lib/kubelet/pods/66478636-93a1-4486-b7bd-65b95ef1d150/volumes" Nov 01 22:01:06.442409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:06.442395 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81363b1-f85c-4d3e-ac08-62aab7b7095d" path="/var/lib/kubelet/pods/f81363b1-f85c-4d3e-ac08-62aab7b7095d/volumes" Nov 01 22:01:06.634223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:06.634186 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:07.352959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:07.352921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:08.043048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:08.043016 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:08.175388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:08.175346 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:09.044690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:09.044627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:09.485172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:09.485064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:09.909924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:09.909892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:10.740706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:10.740638 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:10.869291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:10.869241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:11.740617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:11.740573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:12.026432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:12.026398 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd"] Nov 01 22:01:12.032189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:12.032163 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed9728e-m9zsd"] Nov 01 22:01:12.442842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:12.442755 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3b8031-d0f9-4e4a-b471-6529f62c4502" path="/var/lib/kubelet/pods/fb3b8031-d0f9-4e4a-b471-6529f62c4502/volumes" Nov 01 22:01:12.590169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:12.590131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:13.347065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:13.347022 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:14.060068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:14.060030 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:14.827135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:14.827092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:15.031843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:15.031804 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h"] Nov 01 22:01:15.046940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:15.046905 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e5f2c4-8sd9h"] Nov 01 22:01:15.641355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:15.641313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:15.820507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:15.820470 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:15.988339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:15.988258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:01:16.438629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:16.438597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:16.442026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:16.442000 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f56d51-6e01-4c34-8e19-2fd193b3da6f" path="/var/lib/kubelet/pods/a1f56d51-6e01-4c34-8e19-2fd193b3da6f/volumes" Nov 01 22:01:16.734279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:16.734191 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:16.867690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:16.867652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:17.012811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:17.012716 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:17.033109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:17.033080 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm"] Nov 01 22:01:17.038532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:17.038508 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1fe96-plbpm"] Nov 01 22:01:17.254331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:17.254297 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:17.474788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:17.474710 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:01:18.028758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:18.028721 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf"] Nov 01 22:01:18.080467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:18.072862 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf8d366-hctzf"] Nov 01 22:01:18.189471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:18.184090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:18.466623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:18.466539 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583d41a1-c8d1-4625-86cb-aea64f957630" path="/var/lib/kubelet/pods/583d41a1-c8d1-4625-86cb-aea64f957630/volumes" Nov 01 22:01:18.467096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:18.467081 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4892cee-16ca-4fa2-a0a8-e2178271a588" path="/var/lib/kubelet/pods/a4892cee-16ca-4fa2-a0a8-e2178271a588/volumes" Nov 01 22:01:18.859348 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:18.859311 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:01:19.299400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:19.299363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:20.253109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:20.253073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:21.038594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:21.038563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:22.047352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:22.047310 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:22.181093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:22.181052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:22.741509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:22.741473 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:23.752383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:23.752341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:24.026994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.026965 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp"] Nov 01 22:01:24.032765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.032732 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9ffa0e-xwfsp"] Nov 01 22:01:24.061846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.061809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:24.188518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.188476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:24.320991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.320899 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:24.440967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.440935 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82f3d4c-373e-441f-90ae-f1fda46bf788" path="/var/lib/kubelet/pods/f82f3d4c-373e-441f-90ae-f1fda46bf788/volumes" Nov 01 22:01:24.556821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.556788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:24.681392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:24.681289 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:25.026735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.025041 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:01:25.054440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.054404 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:01:25.054628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.054554 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.058240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.058220 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\"" Nov 01 22:01:25.092886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.092861 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:25.243546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.243516 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-a812fa"] Nov 01 22:01:25.256562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.256534 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:25.268605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.268058 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 22:01:25.329469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.315363 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-a812fa"] Nov 01 22:01:25.330629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.330597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.330859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.330841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.333839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.331039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.334048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.334024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.339544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.339516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.339851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.339783 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtpb\" (UniqueName: \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\") pod \"test-curl-6953-a812fa\" (UID: \"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e\") " pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:25.340056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.340039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.346192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.344797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvr8x\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.346192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.344861 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.346192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.344914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.388250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.387136 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:25.447914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.447689 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.447921 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.447996 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448052 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtpb\" (UniqueName: \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\") pod \"test-curl-6953-a812fa\" (UID: \"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e\") " pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:25.448298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448144 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvr8x\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448310 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.448558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.448539 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.449040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.449011 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.449825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.449796 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.449956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.449938 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.450166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.450129 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.450967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.450946 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.453037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.452985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.455119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.455098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.456157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.456136 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.500502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.493080 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtpb\" (UniqueName: \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\") pod \"test-curl-6953-a812fa\" (UID: \"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e\") " pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:25.532514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.530300 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvr8x\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.535340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.534802 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.566881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.566835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:25.616468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.616133 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:25.671314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.671282 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:25.972856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:25.972383 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-a812fa"] Nov 01 22:01:25.975809 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:01:25.975772 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb1303b2_ceb7_4afc_a59c_a90f47a1cc2e.slice/crio-72e7395f8a4bf50441b8d11606d1b42c381b7af83db29fecbe45b108fe49a0a6 WatchSource:0}: Error finding container 72e7395f8a4bf50441b8d11606d1b42c381b7af83db29fecbe45b108fe49a0a6: Status 404 returned error can't find the container with id 72e7395f8a4bf50441b8d11606d1b42c381b7af83db29fecbe45b108fe49a0a6 Nov 01 22:01:26.108735 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:01:26.108692 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6156be48_32e6_48dd_929a_9b00b1f27c9e.slice/crio-d875d8b8a1d9d27616ce3a2d9a10f856be1ad69151698ea90dd5d99fddcc7c14 WatchSource:0}: Error finding container d875d8b8a1d9d27616ce3a2d9a10f856be1ad69151698ea90dd5d99fddcc7c14: Status 404 returned error can't find the container with id d875d8b8a1d9d27616ce3a2d9a10f856be1ad69151698ea90dd5d99fddcc7c14 Nov 01 22:01:26.110825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.110763 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:01:26.112221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.112180 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:01:26.112316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.112261 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:01:26.112361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.112330 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:01:26.324505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.324397 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:26.376192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.375060 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerStarted","Data":"438831d60906510dea79d69725665b1c3e96f361180cc4999bef26de864f9e94"} Nov 01 22:01:26.376192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.375137 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerStarted","Data":"d875d8b8a1d9d27616ce3a2d9a10f856be1ad69151698ea90dd5d99fddcc7c14"} Nov 01 22:01:26.379344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.379305 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-a812fa" event={"ID":"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e","Type":"ContainerStarted","Data":"84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731"} Nov 01 22:01:26.379520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.379355 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-a812fa" event={"ID":"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e","Type":"ContainerStarted","Data":"72e7395f8a4bf50441b8d11606d1b42c381b7af83db29fecbe45b108fe49a0a6"} Nov 01 22:01:26.413933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.413822 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podStartSLOduration=2.41380107 podStartE2EDuration="2.41380107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:26.398994946 +0000 UTC m=+1040.091975433" watchObservedRunningTime="2026-11-01 22:01:26.41380107 +0000 UTC m=+1040.106781556" Nov 01 22:01:26.429809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.429747 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-a812fa" podStartSLOduration=1.429726864 podStartE2EDuration="1.429726864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:26.423104532 +0000 UTC m=+1040.116085021" watchObservedRunningTime="2026-11-01 22:01:26.429726864 +0000 UTC m=+1040.122707351" Nov 01 22:01:26.671859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:26.671783 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:27.070304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.070263 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:27.132110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.132072 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-a812fa"] Nov 01 22:01:27.146636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.146601 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-a812fa"] Nov 01 22:01:27.300401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.300366 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-55c670"] Nov 01 22:01:27.301197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.301171 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" containerName="test-curl-6953-a812fa" Nov 01 22:01:27.301302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.301198 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" containerName="test-curl-6953-a812fa" Nov 01 22:01:27.301490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.301386 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" containerName="test-curl-6953-a812fa" Nov 01 22:01:27.327173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.327106 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-55c670"] Nov 01 22:01:27.327484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.327400 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:27.393753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.393717 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtz6\" (UniqueName: \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\") pod \"test-curl-6953-55c670\" (UID: \"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba\") " pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:27.495665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.495600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtz6\" (UniqueName: \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\") pod \"test-curl-6953-55c670\" (UID: \"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba\") " pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:27.507730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.507706 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtz6\" (UniqueName: \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\") pod \"test-curl-6953-55c670\" (UID: \"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba\") " pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:27.651888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.651811 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:27.679747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.679720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:27.911299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.911206 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:27.979975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:27.979934 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-55c670"] Nov 01 22:01:28.407307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.407268 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-55c670" event={"ID":"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba","Type":"ContainerStarted","Data":"10b24b5ad8a48563cfb7585c1200ae1841d8dcfbf20656c1a4e1d65ac1955c23"} Nov 01 22:01:28.407795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.407315 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-55c670" event={"ID":"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba","Type":"ContainerStarted","Data":"612f3f6c1f0156200ec24d1e015b5d176552567d34954050eeedc723d6b87c60"} Nov 01 22:01:28.407795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.407568 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-a812fa" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" containerName="test-curl-6953-a812fa" containerID="cri-o://84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731" gracePeriod=2 Nov 01 22:01:28.408362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.408337 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:28.408914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.408891 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:01:28.429478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.429400 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-55c670" podStartSLOduration=1.42937934 podStartE2EDuration="1.42937934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:28.425874391 +0000 UTC m=+1042.118854878" watchObservedRunningTime="2026-11-01 22:01:28.42937934 +0000 UTC m=+1042.122359825" Nov 01 22:01:28.451043 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:28.451010 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-a812fa\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:28.641225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.641187 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:28.776351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.776322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:28.924470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:28.918972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:29.242233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.242150 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-55c670"] Nov 01 22:01:29.249035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.249007 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-55c670"] Nov 01 22:01:29.400576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.400540 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-33e011"] Nov 01 22:01:29.401237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.401221 5036 cpu_manager.go:414] "containerMap: removing container" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" containerName="test-curl-6953-55c670" Nov 01 22:01:29.401237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.401236 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" containerName="test-curl-6953-55c670" Nov 01 22:01:29.401384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.401366 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" containerName="test-curl-6953-55c670" Nov 01 22:01:29.405515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.405489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:29.418466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.414651 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-33e011"] Nov 01 22:01:29.544815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.544777 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sbg\" (UniqueName: \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\") pod \"test-curl-6953-33e011\" (UID: \"3c9646d4-9e12-4278-b28d-2b5a4f9f411b\") " pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:29.646224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.646190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sbg\" (UniqueName: \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\") pod \"test-curl-6953-33e011\" (UID: \"3c9646d4-9e12-4278-b28d-2b5a4f9f411b\") " pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:29.658562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.658534 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sbg\" (UniqueName: \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\") pod \"test-curl-6953-33e011\" (UID: \"3c9646d4-9e12-4278-b28d-2b5a4f9f411b\") " pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:29.733648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.733617 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:29.747515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.747255 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:29.890607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:29.890497 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:30.124985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.124943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:30.221598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.213547 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-33e011"] Nov 01 22:01:30.445748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.445712 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-33e011" event={"ID":"3c9646d4-9e12-4278-b28d-2b5a4f9f411b","Type":"ContainerStarted","Data":"ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919"} Nov 01 22:01:30.445748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.445752 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-33e011" event={"ID":"3c9646d4-9e12-4278-b28d-2b5a4f9f411b","Type":"ContainerStarted","Data":"84cca12786a5c2bf2180f630d6d84b04f9691082dc660f8d4df69602b998b4f6"} Nov 01 22:01:30.446231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.445965 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-55c670" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" containerName="test-curl-6953-55c670" containerID="cri-o://10b24b5ad8a48563cfb7585c1200ae1841d8dcfbf20656c1a4e1d65ac1955c23" gracePeriod=2 Nov 01 22:01:30.466649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.466601 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-33e011" podStartSLOduration=1.466584465 podStartE2EDuration="1.466584465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:30.461438563 +0000 UTC m=+1044.154419049" watchObservedRunningTime="2026-11-01 22:01:30.466584465 +0000 UTC m=+1044.159564950" Nov 01 22:01:30.619602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.619561 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:30.703352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.703323 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\" (UniqueName: \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\") pod \"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e\" (UID: \"cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e\") " Nov 01 22:01:30.706322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.706295 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb" pod "cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" (UID: "cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e"). InnerVolumeSpecName "kube-api-access-bxtpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:30.805774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.805739 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtpb\" (UniqueName: \"kubernetes.io/projected/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e-kube-api-access-bxtpb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:30.836241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:30.836207 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:31.026370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.026335 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8"] Nov 01 22:01:31.032455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.032420 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e546cd-hrvd8"] Nov 01 22:01:31.144657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.144623 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-33e011"] Nov 01 22:01:31.151941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.151674 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-33e011"] Nov 01 22:01:31.293195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.293121 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-2a9ce7"] Nov 01 22:01:31.293871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.293854 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" containerName="test-curl-6953-33e011" Nov 01 22:01:31.293871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.293872 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" containerName="test-curl-6953-33e011" Nov 01 22:01:31.294002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.293993 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" containerName="test-curl-6953-33e011" Nov 01 22:01:31.298566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.298534 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:31.304221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.304193 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-2a9ce7"] Nov 01 22:01:31.314233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.314203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85lr\" (UniqueName: \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\") pod \"test-curl-6953-2a9ce7\" (UID: \"562f30db-914e-4e92-80c6-49307c184eb8\") " pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:31.416270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.416242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85lr\" (UniqueName: \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\") pod \"test-curl-6953-2a9ce7\" (UID: \"562f30db-914e-4e92-80c6-49307c184eb8\") " pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:31.428283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.428254 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85lr\" (UniqueName: \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\") pod \"test-curl-6953-2a9ce7\" (UID: \"562f30db-914e-4e92-80c6-49307c184eb8\") " pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:31.457844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.457817 5036 generic.go:356] "Generic (PLEG): container finished" podID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" containerID="84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731" exitCode=137 Nov 01 22:01:31.458214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.457878 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:31.458214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.457916 5036 scope.go:122] "RemoveContainer" containerID="84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731" Nov 01 22:01:31.469696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.469675 5036 scope.go:122] "RemoveContainer" containerID="84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731" Nov 01 22:01:31.469966 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:31.469947 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731\": container with ID starting with 84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731 not found: ID does not exist" containerID="84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731" Nov 01 22:01:31.470013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.469974 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731"} err="failed to get container status \"84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731\": rpc error: code = NotFound desc = could not find container \"84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731\": container with ID starting with 84a2365ef806ead1cc1be367a9fb94d62484ceb23de2d9e2943b1714be99f731 not found: ID does not exist" Nov 01 22:01:31.470252 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:31.470235 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-a812fa\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" pod="opendatahub/test-curl-6953-a812fa" Nov 01 22:01:31.533910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.533879 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:31.619590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.619508 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:31.936338 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:01:31.936019 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod562f30db_914e_4e92_80c6_49307c184eb8.slice/crio-dda460474793e715daeb2cf337fee86214c01695416e493cdaedb90b6309821b WatchSource:0}: Error finding container dda460474793e715daeb2cf337fee86214c01695416e493cdaedb90b6309821b: Status 404 returned error can't find the container with id dda460474793e715daeb2cf337fee86214c01695416e493cdaedb90b6309821b Nov 01 22:01:31.936338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:31.936328 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-2a9ce7"] Nov 01 22:01:32.328045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.327995 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:32.442924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.442884 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9e2989-45b3-4887-bc38-43c5992e2513" path="/var/lib/kubelet/pods/5b9e2989-45b3-4887-bc38-43c5992e2513/volumes" Nov 01 22:01:32.443530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.443502 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e" path="/var/lib/kubelet/pods/cb1303b2-ceb7-4afc-a59c-a90f47a1cc2e/volumes" Nov 01 22:01:32.475347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.475315 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-2a9ce7" event={"ID":"562f30db-914e-4e92-80c6-49307c184eb8","Type":"ContainerStarted","Data":"f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893"} Nov 01 22:01:32.475743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.475366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-2a9ce7" event={"ID":"562f30db-914e-4e92-80c6-49307c184eb8","Type":"ContainerStarted","Data":"dda460474793e715daeb2cf337fee86214c01695416e493cdaedb90b6309821b"} Nov 01 22:01:32.477264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.477241 5036 generic.go:356] "Generic (PLEG): container finished" podID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" containerID="10b24b5ad8a48563cfb7585c1200ae1841d8dcfbf20656c1a4e1d65ac1955c23" exitCode=137 Nov 01 22:01:32.478776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.478753 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-33e011" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" containerName="test-curl-6953-33e011" containerID="cri-o://ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919" gracePeriod=2 Nov 01 22:01:32.497041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.496996 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-2a9ce7" podStartSLOduration=1.496984307 podStartE2EDuration="1.496984307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:32.491624686 +0000 UTC m=+1046.184605173" watchObservedRunningTime="2026-11-01 22:01:32.496984307 +0000 UTC m=+1046.189964793" Nov 01 22:01:32.641866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.641843 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:32.737591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.737550 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\" (UniqueName: \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\") pod \"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba\" (UID: \"daf9ad98-5730-4bac-a7f2-df4e0d76d7ba\") " Nov 01 22:01:32.739984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.739956 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6" pod "daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" (UID: "daf9ad98-5730-4bac-a7f2-df4e0d76d7ba"). InnerVolumeSpecName "kube-api-access-7xtz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:32.839841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:32.839812 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xtz6\" (UniqueName: \"kubernetes.io/projected/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba-kube-api-access-7xtz6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:33.012959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.012815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:33.173588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.173543 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-2a9ce7"] Nov 01 22:01:33.182052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.182016 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-2a9ce7"] Nov 01 22:01:33.491153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.491125 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:33.491690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.491181 5036 scope.go:122] "RemoveContainer" containerID="10b24b5ad8a48563cfb7585c1200ae1841d8dcfbf20656c1a4e1d65ac1955c23" Nov 01 22:01:33.502377 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:33.502351 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-55c670\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" pod="opendatahub/test-curl-6953-55c670" Nov 01 22:01:33.708852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.708806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:33.854352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.854313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:33.987755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:33.987716 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:34.026194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.026155 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z"] Nov 01 22:01:34.031709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.031684 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7dce8f-k9w4z"] Nov 01 22:01:34.441978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.441938 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f1b363-b6d8-40a9-9369-aa76e109e994" path="/var/lib/kubelet/pods/d4f1b363-b6d8-40a9-9369-aa76e109e994/volumes" Nov 01 22:01:34.442616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.442594 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf9ad98-5730-4bac-a7f2-df4e0d76d7ba" path="/var/lib/kubelet/pods/daf9ad98-5730-4bac-a7f2-df4e0d76d7ba/volumes" Nov 01 22:01:34.462815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.462776 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:34.508338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.508303 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-2a9ce7" podUID="562f30db-914e-4e92-80c6-49307c184eb8" containerName="test-curl-6953-2a9ce7" containerID="cri-o://f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893" gracePeriod=2 Nov 01 22:01:34.745865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.745841 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:34.788224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.788196 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\" (UniqueName: \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\") pod \"3c9646d4-9e12-4278-b28d-2b5a4f9f411b\" (UID: \"3c9646d4-9e12-4278-b28d-2b5a4f9f411b\") " Nov 01 22:01:34.790265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.790233 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg" pod "3c9646d4-9e12-4278-b28d-2b5a4f9f411b" (UID: "3c9646d4-9e12-4278-b28d-2b5a4f9f411b"). InnerVolumeSpecName "kube-api-access-p5sbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:34.889896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:34.889868 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sbg\" (UniqueName: \"kubernetes.io/projected/3c9646d4-9e12-4278-b28d-2b5a4f9f411b-kube-api-access-p5sbg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:35.139682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.139641 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:35.519716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.519685 5036 generic.go:356] "Generic (PLEG): container finished" podID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" containerID="ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919" exitCode=137 Nov 01 22:01:35.520133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.519762 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:35.520133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.519803 5036 scope.go:122] "RemoveContainer" containerID="ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919" Nov 01 22:01:35.531110 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:35.531080 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-33e011\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:35.531985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.531966 5036 scope.go:122] "RemoveContainer" containerID="ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919" Nov 01 22:01:35.532256 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:35.532236 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919\": container with ID starting with ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919 not found: ID does not exist" containerID="ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919" Nov 01 22:01:35.532305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:35.532266 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919"} err="failed to get container status \"ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919\": rpc error: code = NotFound desc = could not find container \"ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919\": container with ID starting with ca75f28a5df5eaea8c877a7ae6074e9bea109433f057c147aa160af38c646919 not found: ID does not exist" Nov 01 22:01:36.020890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.020857 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:36.442218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.442144 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" path="/var/lib/kubelet/pods/3c9646d4-9e12-4278-b28d-2b5a4f9f411b/volumes" Nov 01 22:01:36.446252 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:36.446227 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-33e011\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="3c9646d4-9e12-4278-b28d-2b5a4f9f411b" pod="opendatahub/test-curl-6953-33e011" Nov 01 22:01:36.731012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.730985 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:36.739434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.739403 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:36.823428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.823386 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\" (UniqueName: \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\") pod \"562f30db-914e-4e92-80c6-49307c184eb8\" (UID: \"562f30db-914e-4e92-80c6-49307c184eb8\") " Nov 01 22:01:36.827670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.827633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr" pod "562f30db-914e-4e92-80c6-49307c184eb8" (UID: "562f30db-914e-4e92-80c6-49307c184eb8"). InnerVolumeSpecName "kube-api-access-m85lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:36.926925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:36.926741 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85lr\" (UniqueName: \"kubernetes.io/projected/562f30db-914e-4e92-80c6-49307c184eb8-kube-api-access-m85lr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:01:37.439287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.439250 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:37.561040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.561003 5036 generic.go:356] "Generic (PLEG): container finished" podID="562f30db-914e-4e92-80c6-49307c184eb8" containerID="f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893" exitCode=137 Nov 01 22:01:37.561232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.561053 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:37.561232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.561071 5036 scope.go:122] "RemoveContainer" containerID="f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893" Nov 01 22:01:37.563425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.563361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:37.583904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.583876 5036 scope.go:122] "RemoveContainer" containerID="f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893" Nov 01 22:01:37.584950 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:37.584624 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893\": container with ID starting with f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893 not found: ID does not exist" containerID="f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893" Nov 01 22:01:37.584950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:37.584691 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893"} err="failed to get container status \"f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893\": rpc error: code = NotFound desc = could not find container \"f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893\": container with ID starting with f3c5d53419d2facf21681dd1a03726ea60b32a52905a4e659fa37518437ea893 not found: ID does not exist" Nov 01 22:01:37.588515 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:37.588467 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-2a9ce7\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="562f30db-914e-4e92-80c6-49307c184eb8" pod="opendatahub/test-curl-6953-2a9ce7" Nov 01 22:01:38.329905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:38.329864 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:38.443589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:38.443555 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562f30db-914e-4e92-80c6-49307c184eb8" path="/var/lib/kubelet/pods/562f30db-914e-4e92-80c6-49307c184eb8/volumes" Nov 01 22:01:39.086225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:39.086185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:39.752221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:39.752185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:40.549302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:40.549265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:41.242535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:41.242416 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:41.920080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:41.920035 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:42.026758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:42.026721 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn"] Nov 01 22:01:42.033996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:42.033964 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60c217e-bpwdn"] Nov 01 22:01:42.440067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:42.440035 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4" path="/var/lib/kubelet/pods/43c2e4b8-3bc6-4d51-ae6d-58d62deba0a4/volumes" Nov 01 22:01:42.707248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:42.707138 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:43.452289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:43.452247 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:44.131411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:44.131374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:44.608481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:44.608416 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:44.849994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:44.849961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:44.972364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:44.972274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:45.569188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:45.569150 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:46.358963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:46.358927 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:47.032108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:47.032073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:48.125472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:48.124420 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:48.713236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:48.713193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:48.973319 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:01:48.973180 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:01:49.366482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:49.366405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:50.264355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.264314 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:01:50.265567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.265539 5036 cpu_manager.go:414] "containerMap: removing container" podUID="562f30db-914e-4e92-80c6-49307c184eb8" containerName="test-curl-6953-2a9ce7" Nov 01 22:01:50.265567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.265564 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562f30db-914e-4e92-80c6-49307c184eb8" containerName="test-curl-6953-2a9ce7" Nov 01 22:01:50.265823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.265806 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="562f30db-914e-4e92-80c6-49307c184eb8" containerName="test-curl-6953-2a9ce7" Nov 01 22:01:50.274247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.273004 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.281739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.281694 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:01:50.283776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.283657 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-21dd16de\"/\"e2e-lifecycle-model-21dd16de-kserve-self-signed-certs\"" Nov 01 22:01:50.283776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.283747 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-21dd16de\"/\"default-dockercfg-dd8vd\"" Nov 01 22:01:50.354466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.351123 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-21dd16de\"/\"kube-root-ca.crt\"" Nov 01 22:01:50.354466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.351244 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-21dd16de\"/\"openshift-service-ca.crt\"" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.411547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.411701 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.411813 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.411861 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rps75\" (UniqueName: \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.411920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.412467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.412024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.432600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.421175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.527785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.527890 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.528034 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.533992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.535408 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.542493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.542583 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rps75\" (UniqueName: \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.542656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.551823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.543088 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.561193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.560205 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.602237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.565328 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.645909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.632279 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rps75\" (UniqueName: \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\") pod \"e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.663465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.649350 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:50.776201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:50.776110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:51.542469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.521612 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:01:51.589465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.587550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:51.595468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.595351 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:51.610466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.607580 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:01:51.680933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.680499 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:51.932714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:51.932592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" event={"ID":"4f756862-7959-4a07-b087-4ac08be816b3","Type":"ContainerStarted","Data":"84ba94477b8fb2f4a775da62cba0a1f77d0baa5ffbe49d8e4d0f2666b2f38a2e"} Nov 01 22:01:52.173195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.172541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.280294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.279758 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.444469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.415378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:52.570531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.566929 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:01:52.570531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.567334 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:01:52.589329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.575290 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:01:52.593477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.589681 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.600463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.598837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:01:52.661485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.653707 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:01:52.661485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.653779 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:01:52.661485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.653816 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:01:52.803470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.746546 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.846066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.845983 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:01:52.870854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:52.870822 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:01:53.075144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.061557 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:01:53.217413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.187560 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:01:53.217413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.188127 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:01:53.295469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.283313 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:53.298022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.296739 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:01:53.586493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.573054 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 01 22:01:53.586493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.579048 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:53.588303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.588268 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:01:53.589216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.589197 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:01:53.589429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.589403 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:01:53.750895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.750850 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:53.779065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:53.779035 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:01:54.191472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.187841 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.221472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.215414 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:54.256467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.254090 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:01:54.294476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.294110 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:01:54.338467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.332564 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:01:54.338467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.333660 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.633171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.632761 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.849475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.842286 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.869612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.869559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.993012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.992912 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.998966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:54.996502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:01:55.070916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:55.070887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:01:55.083468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:55.083316 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:01:55.174700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:55.174671 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:01:55.263594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:55.263569 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:01:55.439349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:55.439308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:56.227309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:56.227265 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" event={"ID":"4f756862-7959-4a07-b087-4ac08be816b3","Type":"ContainerStarted","Data":"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8"} Nov 01 22:01:56.249074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:56.249032 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podStartSLOduration=6.249018199 podStartE2EDuration="6.249018199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:56.247429111 +0000 UTC m=+1069.940409598" watchObservedRunningTime="2026-11-01 22:01:56.249018199 +0000 UTC m=+1069.941998685" Nov 01 22:01:56.318875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:56.318835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:56.650033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:56.650001 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:01:57.135692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:57.135660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:57.263675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:57.263644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:57.858746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:57.858711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:58.675791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:58.675745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:58.805483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:58.803600 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:58.805483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:58.803660 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:59.502679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:01:59.502643 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:00.141234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:00.141193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:00.650226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:00.650192 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:02:00.663151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:00.663123 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:02:00.840637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:00.840589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:01.287454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.287422 5036 generic.go:356] "Generic (PLEG): container finished" podID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerID="ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99" exitCode=0 Nov 01 22:02:01.287937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.287500 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerDied","Data":"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99"} Nov 01 22:02:01.287997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.287967 5036 scope.go:122] "RemoveContainer" containerID="ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99" Nov 01 22:02:01.289806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.289774 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:01.289927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.289848 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:01.289927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.289885 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:01.305372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.305339 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:02:01.653766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:01.653718 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:02.313746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:02.313713 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerStarted","Data":"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b"} Nov 01 22:02:02.462868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:02.462837 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:02:02.463037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:02.462880 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:02:02.463037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:02.462938 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 22:02:02.522805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:02.522769 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:03.462344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:03.462307 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:02:03.462802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:03.462375 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 22:02:03.469714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:03.469676 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:04.273820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:04.273784 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:04.462222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:04.462190 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:02:04.462423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:04.462248 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 22:02:05.095472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:05.090989 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:05.462860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:05.462782 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:02:05.462860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:05.462841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 22:02:05.951589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:05.951552 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:06.127144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:06.127103 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:06.462851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:06.462822 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:02:06.463019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:06.462882 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 22:02:06.847707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:06.847663 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:07.509397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.509361 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:02:07.510246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.510214 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:02:07.511863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.511845 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:02:07.520996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.520972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:07.628461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.626842 5036 scope.go:122] "RemoveContainer" containerID="d3619c56333caaabb4d099ddcc95a0bc2efc0b853225e52c7e9506e93db55589" Nov 01 22:02:07.666533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.666510 5036 scope.go:122] "RemoveContainer" containerID="240c24d039f59f6b54cd635ac23f4a3bbc562ef5be4afc085f77bf39d82fdb37" Nov 01 22:02:07.681581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.681551 5036 scope.go:122] "RemoveContainer" containerID="a8681a8108aec5a65669796ea37047cce759bb1d4b06ab45a811f452d77ca49b" Nov 01 22:02:07.708728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.708701 5036 scope.go:122] "RemoveContainer" containerID="5f1b3091dfd771795ba84063351445b3a92f8aad8817aaa312b47162ed0446fa" Nov 01 22:02:07.721084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.721055 5036 scope.go:122] "RemoveContainer" containerID="4037db95ad47f98c10b7745fc3a244a9aac10f4e4b89bc2c9412e64e24a76ab7" Nov 01 22:02:07.731597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.731575 5036 scope.go:122] "RemoveContainer" containerID="012818c82f4fd27e846bb235942c6965dcf10673e950a6459cbbe19e9fcd9071" Nov 01 22:02:07.742800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.742778 5036 scope.go:122] "RemoveContainer" containerID="db5816b10a7f032ccb4b41564f6dbb61a25ce5ba416151a38ce29b87965712dd" Nov 01 22:02:07.753349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.753330 5036 scope.go:122] "RemoveContainer" containerID="656cf6b64e42209c59cd26b0d0cf135e853432e2a3ce106304ae001221bc1c94" Nov 01 22:02:07.765310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.765290 5036 scope.go:122] "RemoveContainer" containerID="1ac9e6a8d0585592ce8a55650eeb7b2bc7e2c18cbe47fc00c09668731ea7e1db" Nov 01 22:02:07.775210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.775182 5036 scope.go:122] "RemoveContainer" containerID="fa4536532b391cf01186e45302c838b9369588eeb843d536814d76070c5b01ac" Nov 01 22:02:07.784957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.784933 5036 scope.go:122] "RemoveContainer" containerID="ecdcd8bdedebd5b347d1ac9994864c38df27598107b8d8fa0abe172799219bdc" Nov 01 22:02:07.795281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:07.795263 5036 scope.go:122] "RemoveContainer" containerID="3fd41024baf72c958c6161072b6eb5511f130edd015d3862de90743d64abf8b2" Nov 01 22:02:08.277196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:08.277154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:08.954831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:08.954787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:09.741767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:09.741732 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:10.439851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:10.439791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:11.223708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:11.223666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:12.037247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:12.037189 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:12.647595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:12.647556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:13.036178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:13.036141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:13.764740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:13.764707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:14.522634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:14.522595 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:15.249737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:15.249698 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:15.920660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:15.920616 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:16.717682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:16.717641 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:17.460291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:17.460253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:18.134046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:18.134004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:18.815344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:18.815309 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:18.879398 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:02:18.876687 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:02:19.191707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:19.191634 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:02:19.799337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:19.799306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:20.850406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:20.850369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:21.545901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:21.545867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:22.028937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:22.028904 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g"] Nov 01 22:02:22.036398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:22.036369 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d23db5a6-x8w6g"] Nov 01 22:02:22.213068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:22.213029 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:22.446036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:22.445956 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ed995b-5678-4339-83a0-0c1e261995f7" path="/var/lib/kubelet/pods/a3ed995b-5678-4339-83a0-0c1e261995f7/volumes" Nov 01 22:02:22.936187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:22.936148 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:23.727309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:23.727268 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:24.450015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:24.449972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:25.025839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:25.025801 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8"] Nov 01 22:02:25.031590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:25.031560 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4afbe64c-62md8"] Nov 01 22:02:25.089945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:25.089912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:25.728145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:25.728102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:26.440938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:26.440907 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd51b98-183c-42be-a6d1-643ec4454d24" path="/var/lib/kubelet/pods/2fd51b98-183c-42be-a6d1-643ec4454d24/volumes" Nov 01 22:02:26.449221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:26.449183 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:27.129629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:27.129581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:27.777726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:27.777672 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:28.449514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:28.440731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:29.151705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:29.151651 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:29.449647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:29.449555 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:02:29.449647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:29.449634 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:02:29.913229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:29.912239 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:30.512482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.512431 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:30.635388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.635363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:30.699899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.699800 5036 generic.go:356] "Generic (PLEG): container finished" podID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerID="438831d60906510dea79d69725665b1c3e96f361180cc4999bef26de864f9e94" exitCode=0 Nov 01 22:02:30.700061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.699888 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerDied","Data":"438831d60906510dea79d69725665b1c3e96f361180cc4999bef26de864f9e94"} Nov 01 22:02:30.700488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.700466 5036 scope.go:122] "RemoveContainer" containerID="438831d60906510dea79d69725665b1c3e96f361180cc4999bef26de864f9e94" Nov 01 22:02:30.702604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.702572 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:30.702676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.702661 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:30.702718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:30.702705 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:31.426815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:31.426773 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:31.720823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:31.720729 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerStarted","Data":"3fcf7c8628a08b535ae105b2810795f8b7465fc9229bd9a24ebb64c64f68136a"} Nov 01 22:02:32.104468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:32.103754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:32.672554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:32.672519 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:02:32.672736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:32.672678 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 01 22:02:32.672781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:32.672729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 01 22:02:32.938330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:32.938224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:33.046463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:33.045467 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp"] Nov 01 22:02:33.046463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:33.046368 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2db6f5-8c5wp"] Nov 01 22:02:33.671895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:33.671861 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 01 22:02:33.672099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:33.671925 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 01 22:02:33.740816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:33.740775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:34.422262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:34.422222 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:34.452488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:34.452435 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9e484a-60b6-4a76-a125-0a5c7835b268" path="/var/lib/kubelet/pods/6b9e484a-60b6-4a76-a125-0a5c7835b268/volumes" Nov 01 22:02:34.672862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:34.672766 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 01 22:02:34.672862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:34.672839 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 01 22:02:35.156978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:35.156941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:35.672716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:35.672684 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 01 22:02:35.673151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:35.672732 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 01 22:02:35.863089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:35.862746 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:36.536354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:36.536316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:36.672558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:36.672527 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 01 22:02:36.672724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:36.672579 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 01 22:02:36.962070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:36.961988 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:37.332434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:37.332400 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:37.676415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:37.676334 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:02:37.677016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:37.676988 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:02:37.677805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:37.677785 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:02:38.112761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:38.112730 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:38.845008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:38.844959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:39.487506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:39.487470 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:39.778856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:39.778817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:40.144680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.144591 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:02:40.166633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.166600 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:02:40.166964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.166936 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.205439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.205404 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b1b5598d-openshift-default-dockercfg-9r76v\"" Nov 01 22:02:40.205875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.205852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:40.292355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.291832 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.292355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.292149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.348578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.348543 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:40.396084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.395994 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.396314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.396294 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.396514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.396499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.396665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.396649 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895dw\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.396762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.396749 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.396887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.396875 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.400574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.400543 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.400841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.400824 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.406462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.401040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.407277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.407250 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.407731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.407708 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508466 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895dw\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508677 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508784 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.509492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.508980 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.511329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.510617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.512234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.512209 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.514077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.513528 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.546667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.545840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.546667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.546590 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895dw\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.557635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.547767 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.558510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.558245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\") pod \"e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:40.582058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:40.581760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:41.010542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.010504 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:02:41.022319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.022277 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:41.022457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.022354 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:41.022457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.022398 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:02:41.029216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.029147 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z"] Nov 01 22:02:41.037859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.037834 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c486a439-d6s2z"] Nov 01 22:02:41.255395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.255309 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:41.848542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.848502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerStarted","Data":"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582"} Nov 01 22:02:41.849013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.848599 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerStarted","Data":"1eb20b7216dfdecb145b937088284d0bb388649803267f165ceb3c641188f692"} Nov 01 22:02:41.879830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.879765 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podStartSLOduration=1.879745755 podStartE2EDuration="1.879745755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:02:41.875265292 +0000 UTC m=+1115.568245783" watchObservedRunningTime="2026-11-01 22:02:41.879745755 +0000 UTC m=+1115.572726242" Nov 01 22:02:41.988729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:41.987389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:42.443125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.443084 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe7304e-4df0-4429-9115-3e9958bbeb0f" path="/var/lib/kubelet/pods/afe7304e-4df0-4429-9115-3e9958bbeb0f/volumes" Nov 01 22:02:42.582638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.582600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:42.588267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.588241 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:42.592655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.592624 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:42.859406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.859310 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:42.860306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:42.860279 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:02:43.027209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:43.027175 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s"] Nov 01 22:02:43.032921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:43.032893 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96db6b56-df79s"] Nov 01 22:02:43.533268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:43.533198 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:44.065108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:44.065074 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:44.191315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:44.191273 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:44.402669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:44.402574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:44.443359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:44.443322 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7faa87e3-4c15-4227-864b-614417a74ba7" path="/var/lib/kubelet/pods/7faa87e3-4c15-4227-864b-614417a74ba7/volumes" Nov 01 22:02:45.150567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:45.150532 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:45.281237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:45.281200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:45.428174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:45.428086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:46.125897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:46.125861 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:46.969330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:46.969277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:47.664195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:47.664098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:48.331044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:48.331009 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 22:02:48.331483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:48.331231 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" containerID="cri-o://5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e" gracePeriod=30 Nov 01 22:02:48.363158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:48.363118 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:48.859869 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:02:48.859834 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:02:49.149669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:49.149577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:50.025733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:50.025696 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:50.648671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:50.648635 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:02:50.648844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:50.648698 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:02:50.809159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:50.809124 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:51.029797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:51.029758 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn"] Nov 01 22:02:51.038383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:51.038352 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt"] Nov 01 22:02:51.045828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:51.045802 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63dbe56e-qprbn"] Nov 01 22:02:51.050609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:51.050585 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1908b23-z44dt"] Nov 01 22:02:51.485050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:51.484972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:52.148925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:52.148851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:52.440354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:52.440282 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda2d64a-ca89-433b-8ff5-89ace6b52474" path="/var/lib/kubelet/pods/bda2d64a-ca89-433b-8ff5-89ace6b52474/volumes" Nov 01 22:02:52.440793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:52.440777 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be56b6f0-adae-4bea-acef-da745eadd701" path="/var/lib/kubelet/pods/be56b6f0-adae-4bea-acef-da745eadd701/volumes" Nov 01 22:02:52.846125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:52.846089 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:53.545057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.545027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:53.675931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.672881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:53.806824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.806799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 22:02:53.869317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.869276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.869697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.869670 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.869814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.869779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.870661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.870631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:02:53.870966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.870947 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:02:53.875458 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.873641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:02:53.971002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.970913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.970974 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971046 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971084 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971134 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971207 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\") pod \"18a17808-2855-4640-9c6f-a41121f028f0\" (UID: \"18a17808-2855-4640-9c6f-a41121f028f0\") " Nov 01 22:02:53.971612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971586 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:02:53.971738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.971710 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:02:53.972062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972009 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:53.972062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972033 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:53.972062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972048 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:53.972062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972063 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a17808-2855-4640-9c6f-a41121f028f0-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:53.972320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972077 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:53.972320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.972178 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:02:53.974065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.974035 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:02:53.974762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.974736 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "kube-api-access-bw5gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:02:53.979600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.979555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo" pod "18a17808-2855-4640-9c6f-a41121f028f0" (UID: "18a17808-2855-4640-9c6f-a41121f028f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:02:53.986177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.986149 5036 generic.go:356] "Generic (PLEG): container finished" podID="18a17808-2855-4640-9c6f-a41121f028f0" containerID="5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e" exitCode=0 Nov 01 22:02:53.986290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.986212 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" Nov 01 22:02:53.986290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.986228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" event={"ID":"18a17808-2855-4640-9c6f-a41121f028f0","Type":"ContainerDied","Data":"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e"} Nov 01 22:02:53.986416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.986307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld" event={"ID":"18a17808-2855-4640-9c6f-a41121f028f0","Type":"ContainerDied","Data":"cf64d1dd211bfe94e7e6845968db7323cc21c4f120df3fdc194a4feda08a1df7"} Nov 01 22:02:53.986416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:53.986337 5036 scope.go:122] "RemoveContainer" containerID="5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e" Nov 01 22:02:54.003610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.003576 5036 scope.go:122] "RemoveContainer" containerID="5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e" Nov 01 22:02:54.006622 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:02:54.006584 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e\": container with ID starting with 5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e not found: ID does not exist" containerID="5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e" Nov 01 22:02:54.006742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.006635 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e"} err="failed to get container status \"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e\": rpc error: code = NotFound desc = could not find container \"5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e\": container with ID starting with 5fffd3fa31eb46a76df42bf52d32692727e562c9e289360b1d4e3e0aaceffb6e not found: ID does not exist" Nov 01 22:02:54.010224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.010203 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:54.022503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.022477 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 22:02:54.028235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.028214 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-j57ld"] Nov 01 22:02:54.082471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.076342 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a17808-2855-4640-9c6f-a41121f028f0-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:54.082471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.076387 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a17808-2855-4640-9c6f-a41121f028f0-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:54.082471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.076405 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw5gw\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-kube-api-access-bw5gw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:54.082471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.076421 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a17808-2855-4640-9c6f-a41121f028f0-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:02:54.439991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.439958 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a17808-2855-4640-9c6f-a41121f028f0" path="/var/lib/kubelet/pods/18a17808-2855-4640-9c6f-a41121f028f0/volumes" Nov 01 22:02:54.549343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:54.549308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:55.431273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:55.431241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:56.162639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:56.162603 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:56.880932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:56.880891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:57.313863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:57.313829 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:57.641471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:57.641366 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:58.259575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.259495 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm"] Nov 01 22:02:58.260216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.260201 5036 cpu_manager.go:414] "containerMap: removing container" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" Nov 01 22:02:58.260261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.260217 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" Nov 01 22:02:58.260371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.260362 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a17808-2855-4640-9c6f-a41121f028f0" containerName="istio-proxy" Nov 01 22:02:58.265967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.265853 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.271500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.270148 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:02:58.271653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.271560 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:02:58.271729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.271654 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:02:58.271729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.271695 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:02:58.273436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.273413 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm"] Nov 01 22:02:58.365726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.365692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.366141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.365737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wfb\" (UniqueName: \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.366141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.365858 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.383240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.383203 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:58.468148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.468109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.468341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.468184 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wfb\" (UniqueName: \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.468341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.468264 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.469089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.469062 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.471760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.471734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.481716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.481685 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wfb\" (UniqueName: \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\") pod \"maas-api-revoke-sub-d09db57e-8kbpm\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.588605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.588570 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:02:58.876800 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:02:58.876768 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e1c4ac_59a2_4cf4_80a0_d0b58d9a8866.slice/crio-41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc WatchSource:0}: Error finding container 41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc: Status 404 returned error can't find the container with id 41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc Nov 01 22:02:58.877774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:58.877747 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm"] Nov 01 22:02:59.036888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:59.036852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:59.048555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:59.048520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" event={"ID":"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866","Type":"ContainerStarted","Data":"41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc"} Nov 01 22:02:59.709100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:02:59.708992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:00.060078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:00.060049 5036 generic.go:356] "Generic (PLEG): container finished" podID="e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" containerID="37b9a1a54d3c0b37bc0b6155e769a6b238be8a432c215eff9efee6ad51cb23dc" exitCode=0 Nov 01 22:03:00.060253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:00.060140 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" event={"ID":"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866","Type":"ContainerDied","Data":"37b9a1a54d3c0b37bc0b6155e769a6b238be8a432c215eff9efee6ad51cb23dc"} Nov 01 22:03:00.423688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:00.423605 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:01.155369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.155335 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:01.312869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.312840 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:03:01.418147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.418075 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\") pod \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " Nov 01 22:03:01.418147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.418136 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\") pod \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " Nov 01 22:03:01.418348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.418165 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\" (UniqueName: \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\") pod \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\" (UID: \"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866\") " Nov 01 22:03:01.418522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.418493 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca" pod "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" (UID: "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:01.418780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.418757 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:01.420488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.420461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb" pod "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" (UID: "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866"). InnerVolumeSpecName "kube-api-access-s2wfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:01.420584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.420513 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca" pod "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" (UID: "e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:03:01.521038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.521008 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:01.521038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.521040 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wfb\" (UniqueName: \"kubernetes.io/projected/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866-kube-api-access-s2wfb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:01.863873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:01.863836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:02.083989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:02.083955 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" Nov 01 22:03:02.083989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:02.083974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm" event={"ID":"e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866","Type":"ContainerDied","Data":"41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc"} Nov 01 22:03:02.084229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:02.084024 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f7ab678cc66836a9b1091cadcc0b541ecbafa58363d81bf8f0255d48a60dfc" Nov 01 22:03:02.529202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:02.529167 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:03.232404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:03.232367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:04.054607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:04.054565 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:05.221123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:05.221083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:06.032455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:06.032406 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:06.668025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:06.667983 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:07.026113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.026080 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79"] Nov 01 22:03:07.031222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.031195 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e56254-tsq79"] Nov 01 22:03:07.363142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.363052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:07.960581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.960552 5036 scope.go:122] "RemoveContainer" containerID="636a9602f9686bcc2e3e5273a57dde439492492457f2e8ec583c3fd1fcc21570" Nov 01 22:03:07.972381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.972357 5036 scope.go:122] "RemoveContainer" containerID="06b6f1cf3e8f9d3b69b7a565aa2919330c7b0120a09d039e725a97e37b81a4ef" Nov 01 22:03:07.987628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:07.987603 5036 scope.go:122] "RemoveContainer" containerID="fb04f8b4c3933be736d2b08f6e1d3a44c38aa7174f93b76330e6bd3a9d0deb53" Nov 01 22:03:08.002719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.002685 5036 scope.go:122] "RemoveContainer" containerID="857b1525d88658a750a82e7c0324cd93e9cc9636eb85750878b1faab199d7676" Nov 01 22:03:08.017645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.017625 5036 scope.go:122] "RemoveContainer" containerID="f4eece0212aa90796cb9aad28672431e1bc375e2f86fb6eaae690a6fb05e1a8e" Nov 01 22:03:08.029397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.029377 5036 scope.go:122] "RemoveContainer" containerID="cef4e244f9fe6734de52a6b7e368d6d8b5b1381b0f4520591bf1a46f13acfc24" Nov 01 22:03:08.030170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.030150 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9"] Nov 01 22:03:08.036283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.036257 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ee82a6-q8sg9"] Nov 01 22:03:08.039304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.039278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:08.040214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.040198 5036 scope.go:122] "RemoveContainer" containerID="11a34a1a936d77f2c7b64a47b1ee0f693f0470cc2360df24dcc9afb39f599dc9" Nov 01 22:03:08.442498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.442436 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660aaf4f-4ed4-4780-8571-9727560319bb" path="/var/lib/kubelet/pods/660aaf4f-4ed4-4780-8571-9727560319bb/volumes" Nov 01 22:03:08.443008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.442991 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787a27a9-8bce-4235-b2f3-f865827ecd90" path="/var/lib/kubelet/pods/787a27a9-8bce-4235-b2f3-f865827ecd90/volumes" Nov 01 22:03:08.527313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.527274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:08.740879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:08.740793 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:09.029053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:09.029022 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n"] Nov 01 22:03:09.033917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:09.033892 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49fdb93-xkq2n"] Nov 01 22:03:09.277939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:09.277896 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:09.604228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:09.604187 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:10.126892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:10.126856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:10.252549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:10.252510 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:10.397232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:10.397144 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:10.445589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:10.445548 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f467320f-9bc7-47a4-962c-1769d84a1aae" path="/var/lib/kubelet/pods/f467320f-9bc7-47a4-962c-1769d84a1aae/volumes" Nov 01 22:03:10.684834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:10.684745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:11.039233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:11.039197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:11.699314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:11.699279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:12.432308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:12.432271 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:13.172504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:13.171859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:13.751668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:13.751625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:14.533383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:14.533352 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:14.916043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:14.915945 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:15.239488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:15.239353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:15.911407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:15.911356 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:16.652374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:16.652339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:16.851024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:16.850978 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:17.328824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:17.328788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:18.109985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:18.109942 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:18.578341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:18.578301 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:03:18.813906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:18.813864 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:18.887059 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:18.886975 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:03:19.040374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:19.040299 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q"] Nov 01 22:03:19.044993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:19.044960 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c93d125-mrd7q"] Nov 01 22:03:19.668468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:19.661995 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:20.439432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:20.439393 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f983c2-966e-4ea4-9bef-93b99809a331" path="/var/lib/kubelet/pods/21f983c2-966e-4ea4-9bef-93b99809a331/volumes" Nov 01 22:03:20.672096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:20.672058 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:21.037884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:21.037845 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w"] Nov 01 22:03:21.040668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:21.040639 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx"] Nov 01 22:03:21.045152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:21.045129 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7173595-49f5w"] Nov 01 22:03:21.050023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:21.049657 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec90963f-5mclx"] Nov 01 22:03:21.552273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:21.552231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:22.350435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:22.350361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:22.472393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:22.472361 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9dc5fa1-26e6-44bc-ad10-27c44261e527" path="/var/lib/kubelet/pods/a9dc5fa1-26e6-44bc-ad10-27c44261e527/volumes" Nov 01 22:03:22.472913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:22.472888 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea941d9-5123-4441-886b-138af23c6b5f" path="/var/lib/kubelet/pods/eea941d9-5123-4441-886b-138af23c6b5f/volumes" Nov 01 22:03:23.114491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:23.113199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:23.826525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:23.826485 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:24.459775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:24.459738 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:25.289467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:25.286269 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:25.954790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:25.954753 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:26.635514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:26.635343 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:27.334760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:27.334725 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:28.064069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:28.064022 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:28.725851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:28.725699 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:28.901665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:28.901609 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:28.901876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:28.901684 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:29.696268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:29.696219 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:30.414204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.414124 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:30.459748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.459641 5036 generic.go:356] "Generic (PLEG): container finished" podID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerID="e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582" exitCode=0 Nov 01 22:03:30.459748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.459695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerDied","Data":"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582"} Nov 01 22:03:30.460267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.460248 5036 scope.go:122] "RemoveContainer" containerID="e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582" Nov 01 22:03:30.464571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.464531 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:30.464696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.464659 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:30.464740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:30.464725 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:31.141601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.141542 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:31.470920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.470838 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerStarted","Data":"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70"} Nov 01 22:03:31.582883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.582848 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:03:31.583126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.583029 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:31.583126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.583080 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:31.863651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:31.863618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:32.027858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.027824 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb"] Nov 01 22:03:32.032893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.032868 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a536cbc-nkjhb"] Nov 01 22:03:32.440786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.440751 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7bed0f-3fdd-483d-99c5-f47ebfc932b9" path="/var/lib/kubelet/pods/7f7bed0f-3fdd-483d-99c5-f47ebfc932b9/volumes" Nov 01 22:03:32.533690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.533646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:32.582993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.582958 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:32.583263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:32.583234 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:33.162182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:33.162143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:33.598473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:33.598418 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:33.602617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:33.602579 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:33.833549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:33.833500 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:34.548501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:34.548462 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:34.584527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:34.582890 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:34.584527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:34.582962 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:35.317297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:35.317258 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:35.594517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:35.586800 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:35.594517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:35.586871 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:36.029626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.029596 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7"] Nov 01 22:03:36.036565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.036536 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9285a513-lgzr7"] Nov 01 22:03:36.038494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.038464 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:36.041271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.041248 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc"] Nov 01 22:03:36.046486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.046431 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f26d954-rb8qc"] Nov 01 22:03:36.441163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.441087 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9666743e-59a5-4d31-83db-ea9831ccf2db" path="/var/lib/kubelet/pods/9666743e-59a5-4d31-83db-ea9831ccf2db/volumes" Nov 01 22:03:36.441649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.441633 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb4413e-e309-401b-b225-bc06f442eee5" path="/var/lib/kubelet/pods/abb4413e-e309-401b-b225-bc06f442eee5/volumes" Nov 01 22:03:36.460587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.460561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:36.599466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.596807 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:36.599466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.596877 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:36.722426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.722331 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6956-b9d4ce"] Nov 01 22:03:36.723133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.723110 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" containerName="revoke-keys" Nov 01 22:03:36.723249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.723133 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" containerName="revoke-keys" Nov 01 22:03:36.723312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.723290 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" containerName="revoke-keys" Nov 01 22:03:36.728931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.728905 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:36.735972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.735949 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 22:03:36.745136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.745107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:36.750292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.750265 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6956-b9d4ce"] Nov 01 22:03:36.755399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.755077 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\") pod \"test-curl-6956-b9d4ce\" (UID: \"1cfb96b5-4925-4e29-95db-bd9788833ee5\") " pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:36.838500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.838304 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:03:36.844994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.844967 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.872751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.872707 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.872936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.872834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.872999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.872949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.872992 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873028 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873113 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67mh\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\") pod \"test-curl-6956-b9d4ce\" (UID: \"1cfb96b5-4925-4e29-95db-bd9788833ee5\") " pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:36.873280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.873772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.873597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:36.960484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.960428 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:03:36.966324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.966293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:36.968693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:36.968670 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d506c6-openshift-default-dockercfg-wx25k\"" Nov 01 22:03:37.021804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.012042 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\") pod \"test-curl-6956-b9d4ce\" (UID: \"1cfb96b5-4925-4e29-95db-bd9788833ee5\") " pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.074941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075141 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075172 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075205 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67mh\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075307 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075500 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.079695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.075605 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.187469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.132985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.291976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.211646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.291976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.166396 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:37.381646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.172047 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.381897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.205120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.382023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.205524 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.382124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.142637 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.467467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.459008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:37.467467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.460106 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.467467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.463950 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.491955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.491922 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5"] Nov 01 22:03:37.509589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.509553 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67mh\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\") pod \"e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.517302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.517248 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs"] Nov 01 22:03:37.527966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.527901 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ee1dfe7-bwrgs"] Nov 01 22:03:37.530497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.530435 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24587cf-h2np5"] Nov 01 22:03:37.592221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.592184 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 22:03:37.592873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.592261 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 22:03:37.649040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.647922 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:37.800985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.800621 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:37.824605 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:03:37.824563 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cfb96b5_4925_4e29_95db_bd9788833ee5.slice/crio-24a20b502d08e109b88d30ea226128e79ebc27d70b02dd4a37dd4d8d3c746e30 WatchSource:0}: Error finding container 24a20b502d08e109b88d30ea226128e79ebc27d70b02dd4a37dd4d8d3c746e30: Status 404 returned error can't find the container with id 24a20b502d08e109b88d30ea226128e79ebc27d70b02dd4a37dd4d8d3c746e30 Nov 01 22:03:37.829022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:37.829001 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6956-b9d4ce"] Nov 01 22:03:38.125467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.124611 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:03:38.125467 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:03:38.125057 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded15519a_d1c2_4733_9182_9cdedf40f833.slice/crio-dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c WatchSource:0}: Error finding container dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c: Status 404 returned error can't find the container with id dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c Nov 01 22:03:38.128276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.128236 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:38.128387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.128323 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:38.128387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.128370 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:03:38.296654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.293775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:38.451578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.450593 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e0e9ff-417a-4016-be1c-3921683d532c" path="/var/lib/kubelet/pods/65e0e9ff-417a-4016-be1c-3921683d532c/volumes" Nov 01 22:03:38.451578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.451346 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7" path="/var/lib/kubelet/pods/d7a3e3cf-2e6e-4ab6-aa0d-633c0063d7d7/volumes" Nov 01 22:03:38.579744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.579132 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerStarted","Data":"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef"} Nov 01 22:03:38.579744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.579199 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerStarted","Data":"dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c"} Nov 01 22:03:38.582869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.582835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6956-b9d4ce" event={"ID":"1cfb96b5-4925-4e29-95db-bd9788833ee5","Type":"ContainerStarted","Data":"779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27"} Nov 01 22:03:38.583017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.582882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6956-b9d4ce" event={"ID":"1cfb96b5-4925-4e29-95db-bd9788833ee5","Type":"ContainerStarted","Data":"24a20b502d08e109b88d30ea226128e79ebc27d70b02dd4a37dd4d8d3c746e30"} Nov 01 22:03:38.591825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.591802 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:03:38.592982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.592963 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:03:38.593833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.593818 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:03:38.610674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.610621 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podStartSLOduration=2.610605379 podStartE2EDuration="2.610605379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:38.602475473 +0000 UTC m=+1172.295455960" watchObservedRunningTime="2026-11-01 22:03:38.610605379 +0000 UTC m=+1172.303585866" Nov 01 22:03:38.625389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.625338 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6956-b9d4ce" podStartSLOduration=2.625321991 podStartE2EDuration="2.625321991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:38.618673421 +0000 UTC m=+1172.311653908" watchObservedRunningTime="2026-11-01 22:03:38.625321991 +0000 UTC m=+1172.318302481" Nov 01 22:03:38.811466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:38.806570 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:39.158555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.158171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:39.368205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.368168 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6956-b9d4ce"] Nov 01 22:03:39.374323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.374289 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6956-b9d4ce"] Nov 01 22:03:39.806912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.806876 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:39.933910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.933877 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj"] Nov 01 22:03:39.934592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.934576 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" containerName="test-curl-6956-b9d4ce" Nov 01 22:03:39.934650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.934593 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" containerName="test-curl-6956-b9d4ce" Nov 01 22:03:39.934746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.934737 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" containerName="test-curl-6956-b9d4ce" Nov 01 22:03:39.939796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.939778 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:39.947705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.947680 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:03:39.948347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.948326 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:03:39.949392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:39.949371 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj"] Nov 01 22:03:40.015296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.015267 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.015464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.015329 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.015516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.015478 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zpj\" (UniqueName: \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.040841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.040806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:40.043239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.043220 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:03:40.043333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.043267 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:03:40.117795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.117702 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zpj\" (UniqueName: \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.118040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.118016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.118132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.118087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.118923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.118901 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.121005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.120985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.135223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.135176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zpj\" (UniqueName: \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.264154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.264118 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:40.564390 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:03:40.564344 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0635d7c5_1911_4977_801b_78a52c4ae31a.slice/crio-8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55 WatchSource:0}: Error finding container 8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55: Status 404 returned error can't find the container with id 8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55 Nov 01 22:03:40.566877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.566853 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj"] Nov 01 22:03:40.608578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.608540 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" event={"ID":"0635d7c5-1911-4977-801b-78a52c4ae31a","Type":"ContainerStarted","Data":"8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55"} Nov 01 22:03:40.608868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.608836 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6956-b9d4ce" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" containerName="test-curl-6956-b9d4ce" containerID="cri-o://779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27" gracePeriod=2 Nov 01 22:03:40.608978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.608914 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:40.610647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.610626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:03:40.810056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:40.810015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:41.019109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:41.019079 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:41.225241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:41.225146 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:41.621908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:41.621870 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" event={"ID":"0635d7c5-1911-4977-801b-78a52c4ae31a","Type":"ContainerStarted","Data":"2a438d18d8f2ac4a336b758931be4373719d22e057d4898780c6df255572988a"} Nov 01 22:03:41.642738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:41.642685 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" podStartSLOduration=2.642670859 podStartE2EDuration="2.642670859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:41.639859265 +0000 UTC m=+1175.332839752" watchObservedRunningTime="2026-11-01 22:03:41.642670859 +0000 UTC m=+1175.335651345" Nov 01 22:03:42.906229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:42.906119 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:43.005888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.005854 5036 generic.go:356] "Generic (PLEG): container finished" podID="0635d7c5-1911-4977-801b-78a52c4ae31a" containerID="2a438d18d8f2ac4a336b758931be4373719d22e057d4898780c6df255572988a" exitCode=0 Nov 01 22:03:43.007395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.007366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" event={"ID":"0635d7c5-1911-4977-801b-78a52c4ae31a","Type":"ContainerDied","Data":"2a438d18d8f2ac4a336b758931be4373719d22e057d4898780c6df255572988a"} Nov 01 22:03:43.037470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.036645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:43.534577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.529626 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:43.534577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.529705 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:43.667466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.666806 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:43.776213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.776174 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\") pod \"1cfb96b5-4925-4e29-95db-bd9788833ee5\" (UID: \"1cfb96b5-4925-4e29-95db-bd9788833ee5\") " Nov 01 22:03:43.802468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.801976 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb" pod "1cfb96b5-4925-4e29-95db-bd9788833ee5" (UID: "1cfb96b5-4925-4e29-95db-bd9788833ee5"). InnerVolumeSpecName "kube-api-access-dhcbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:43.899520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:43.891105 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/1cfb96b5-4925-4e29-95db-bd9788833ee5-kube-api-access-dhcbb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:44.050682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.050622 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml"] Nov 01 22:03:44.059479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.059430 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49287ea-txnml"] Nov 01 22:03:44.071460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.067729 5036 generic.go:356] "Generic (PLEG): container finished" podID="1cfb96b5-4925-4e29-95db-bd9788833ee5" containerID="779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27" exitCode=137 Nov 01 22:03:44.071460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.068090 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:44.076403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.076064 5036 scope.go:122] "RemoveContainer" containerID="779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27" Nov 01 22:03:44.089463 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:44.086767 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6956-b9d4ce\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:44.107803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.106770 5036 scope.go:122] "RemoveContainer" containerID="779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27" Nov 01 22:03:44.107803 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:44.107125 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27\": container with ID starting with 779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27 not found: ID does not exist" containerID="779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27" Nov 01 22:03:44.107803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.107163 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27"} err="failed to get container status \"779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27\": rpc error: code = NotFound desc = could not find container \"779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27\": container with ID starting with 779acbf17bdb6572197d2027a6d9fadbd6b59c4e9e2646b2d5bc350874c58b27 not found: ID does not exist" Nov 01 22:03:44.263306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.262912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:44.321428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.321400 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:44.324086 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:44.324048 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6956-b9d4ce\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" pod="opendatahub/test-curl-6956-b9d4ce" Nov 01 22:03:44.410556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.407629 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:44.418491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.415848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\") pod \"0635d7c5-1911-4977-801b-78a52c4ae31a\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " Nov 01 22:03:44.418491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.416017 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\" (UniqueName: \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\") pod \"0635d7c5-1911-4977-801b-78a52c4ae31a\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " Nov 01 22:03:44.418491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.416145 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\") pod \"0635d7c5-1911-4977-801b-78a52c4ae31a\" (UID: \"0635d7c5-1911-4977-801b-78a52c4ae31a\") " Nov 01 22:03:44.418491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.417783 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca" pod "0635d7c5-1911-4977-801b-78a52c4ae31a" (UID: "0635d7c5-1911-4977-801b-78a52c4ae31a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:44.471466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.461655 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca" pod "0635d7c5-1911-4977-801b-78a52c4ae31a" (UID: "0635d7c5-1911-4977-801b-78a52c4ae31a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:03:44.492469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.472956 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfb96b5-4925-4e29-95db-bd9788833ee5" path="/var/lib/kubelet/pods/1cfb96b5-4925-4e29-95db-bd9788833ee5/volumes" Nov 01 22:03:44.492469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.473848 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3881852-34f9-423b-9054-eb18ba2c78bc" path="/var/lib/kubelet/pods/f3881852-34f9-423b-9054-eb18ba2c78bc/volumes" Nov 01 22:03:44.506473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.504633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj" pod "0635d7c5-1911-4977-801b-78a52c4ae31a" (UID: "0635d7c5-1911-4977-801b-78a52c4ae31a"). InnerVolumeSpecName "kube-api-access-d8zpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:44.534472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.525141 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:44.534472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.525179 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zpj\" (UniqueName: \"kubernetes.io/projected/0635d7c5-1911-4977-801b-78a52c4ae31a-kube-api-access-d8zpj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:44.534472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.525195 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0635d7c5-1911-4977-801b-78a52c4ae31a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:44.559464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:44.551144 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:45.100508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.100475 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" Nov 01 22:03:45.100999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.100517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj" event={"ID":"0635d7c5-1911-4977-801b-78a52c4ae31a","Type":"ContainerDied","Data":"8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55"} Nov 01 22:03:45.100999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.100601 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b1fe0e246e4acb4fc05c7010066227e1c40e54f1296638d821496bf6d5d7d55" Nov 01 22:03:45.513693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.513614 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:45.639469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.638533 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:45.786626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.786588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:45.915627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:45.915586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:46.296297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:46.296250 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:46.470318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:46.470280 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:47.028209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.028166 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm"] Nov 01 22:03:47.033649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.033618 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4"] Nov 01 22:03:47.038088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.038063 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa14e88-j5ltm"] Nov 01 22:03:47.044538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.044513 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d186c4bd-zg6n4"] Nov 01 22:03:47.118860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.118824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:47.160597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.160557 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh"] Nov 01 22:03:47.161948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.161796 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0635d7c5-1911-4977-801b-78a52c4ae31a" containerName="revoke-keys" Nov 01 22:03:47.161948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.161822 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0635d7c5-1911-4977-801b-78a52c4ae31a" containerName="revoke-keys" Nov 01 22:03:47.162264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.161977 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0635d7c5-1911-4977-801b-78a52c4ae31a" containerName="revoke-keys" Nov 01 22:03:47.167847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.167821 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.172341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.172317 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:03:47.172536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.172383 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:03:47.172595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.172576 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:03:47.172646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.172587 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:03:47.178812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.178787 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh"] Nov 01 22:03:47.284889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.284807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.284889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.284849 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.285263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.285240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zhb4\" (UniqueName: \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.386984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.386952 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zhb4\" (UniqueName: \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.387528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.387046 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.387528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.387087 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.388366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.388334 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.389958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.389926 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.399528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.399503 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zhb4\" (UniqueName: \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\") pod \"maas-api-revoke-sub-98b4a743-wtssh\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.500233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.500195 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:47.802733 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:03:47.802701 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3c14cb_ce0d_4f87_bfa7_c0b954be71c4.slice/crio-3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494 WatchSource:0}: Error finding container 3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494: Status 404 returned error can't find the container with id 3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494 Nov 01 22:03:47.803885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.803860 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh"] Nov 01 22:03:47.937662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:47.937580 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:48.156236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.156200 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" event={"ID":"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4","Type":"ContainerStarted","Data":"3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494"} Nov 01 22:03:48.444812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.444720 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff04bab-ff86-4df4-989b-9357fbbccbde" path="/var/lib/kubelet/pods/0ff04bab-ff86-4df4-989b-9357fbbccbde/volumes" Nov 01 22:03:48.445649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.445622 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c8ac64-a338-4cff-930c-1fcf33cd192d" path="/var/lib/kubelet/pods/d0c8ac64-a338-4cff-930c-1fcf33cd192d/volumes" Nov 01 22:03:48.622068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.622029 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 22:03:48.622310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.622266 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" containerID="cri-o://57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc" gracePeriod=30 Nov 01 22:03:48.658313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:48.658258 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:48.865147 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:48.865106 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:03:49.167646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:49.167554 5036 generic.go:356] "Generic (PLEG): container finished" podID="ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" containerID="18696a4b9ec23bcbf93084fa726e8131867c4264f92f2dbd1f901fff8da0c51c" exitCode=0 Nov 01 22:03:49.167805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:49.167640 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" event={"ID":"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4","Type":"ContainerDied","Data":"18696a4b9ec23bcbf93084fa726e8131867c4264f92f2dbd1f901fff8da0c51c"} Nov 01 22:03:49.443045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:49.442952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:50.174144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.174107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:50.320518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.309906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:50.541502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.532617 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:50.605824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.605779 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:50.714524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.714154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\" (UniqueName: \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\") pod \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " Nov 01 22:03:50.714524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.714343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\") pod \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " Nov 01 22:03:50.714524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.714380 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\") pod \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\" (UID: \"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4\") " Nov 01 22:03:50.721571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.717220 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca" pod "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" (UID: "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:50.747272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.747227 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4" pod "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" (UID: "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4"). InnerVolumeSpecName "kube-api-access-4zhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:50.748796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.748770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca" pod "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" (UID: "ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:03:50.816292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.816257 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zhb4\" (UniqueName: \"kubernetes.io/projected/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-kube-api-access-4zhb4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:50.816292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.816289 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:50.816580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:50.816304 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:51.041617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.041575 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4"] Nov 01 22:03:51.046572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.046544 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9cc98-b4xg4"] Nov 01 22:03:51.191018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.190993 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" Nov 01 22:03:51.191018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.191002 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh" event={"ID":"ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4","Type":"ContainerDied","Data":"3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494"} Nov 01 22:03:51.191422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.191072 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3125cf8f3c5ae89618d1e06387b74d2113c8603e7aa9b5b56fdc4d68f9fd2494" Nov 01 22:03:51.492402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:51.492314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:52.441152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:52.441118 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5218e058-1d92-4f50-9327-711436d97642" path="/var/lib/kubelet/pods/5218e058-1d92-4f50-9327-711436d97642/volumes" Nov 01 22:03:53.220511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.220477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:53.432616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.432585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:53.932824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.932799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 22:03:53.977724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.977690 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978186 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978267 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978405 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978535 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978587 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978634 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978672 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\") pod \"20c5c171-9539-4e97-994e-23751c85579f\" (UID: \"20c5c171-9539-4e97-994e-23751c85579f\") " Nov 01 22:03:53.978804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.978780 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:53.979578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.979550 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:53.979578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.979563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:53.979794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.979773 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:53.979889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.979860 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:53.979946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.979932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:53.981688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.980421 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:03:53.984140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.984110 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:53.993905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.993868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "kube-api-access-wcgh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:53.995840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:53.995811 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token" pod "20c5c171-9539-4e97-994e-23751c85579f" (UID: "20c5c171-9539-4e97-994e-23751c85579f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:54.033510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.033481 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc"] Nov 01 22:03:54.037996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.037975 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536f6e79-v4kjc"] Nov 01 22:03:54.080932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080892 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.080932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080931 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c5c171-9539-4e97-994e-23751c85579f-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080946 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080960 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080975 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.080988 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c5c171-9539-4e97-994e-23751c85579f-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.081002 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c5c171-9539-4e97-994e-23751c85579f-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.081182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.081017 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgh4\" (UniqueName: \"kubernetes.io/projected/20c5c171-9539-4e97-994e-23751c85579f-kube-api-access-wcgh4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:54.219087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.218959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:54.222983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.222959 5036 generic.go:356] "Generic (PLEG): container finished" podID="20c5c171-9539-4e97-994e-23751c85579f" containerID="57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc" exitCode=0 Nov 01 22:03:54.223142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.223012 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" Nov 01 22:03:54.223142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.223044 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" event={"ID":"20c5c171-9539-4e97-994e-23751c85579f","Type":"ContainerDied","Data":"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc"} Nov 01 22:03:54.223142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.223104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk" event={"ID":"20c5c171-9539-4e97-994e-23751c85579f","Type":"ContainerDied","Data":"62b1472c4e3ee34b36a813c40b51fb2ad861f96e433f2d4d826b6e2fc2b961e5"} Nov 01 22:03:54.223142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.223136 5036 scope.go:122] "RemoveContainer" containerID="57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc" Nov 01 22:03:54.241906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.241541 5036 scope.go:122] "RemoveContainer" containerID="57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc" Nov 01 22:03:54.242640 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:03:54.242160 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc\": container with ID starting with 57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc not found: ID does not exist" containerID="57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc" Nov 01 22:03:54.242640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.242200 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc"} err="failed to get container status \"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc\": rpc error: code = NotFound desc = could not find container \"57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc\": container with ID starting with 57cad32e68d0f77ed72c389af3c70083ae9f21b97b6d00b170ca42b2324b1cbc not found: ID does not exist" Nov 01 22:03:54.269607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.269549 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 22:03:54.285089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.285051 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-n7zmk"] Nov 01 22:03:54.439542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.439513 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e" path="/var/lib/kubelet/pods/1d0a394e-e5f5-4b0b-9a4c-6b07f0a8153e/volumes" Nov 01 22:03:54.440069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.440056 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c5c171-9539-4e97-994e-23751c85579f" path="/var/lib/kubelet/pods/20c5c171-9539-4e97-994e-23751c85579f/volumes" Nov 01 22:03:54.912508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:54.912468 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:55.054529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:55.054482 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:55.946090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:55.946052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:56.645523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:56.645492 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:57.320278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:57.320242 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:57.451313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:57.451275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:57.582696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:57.582594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:58.021629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.021577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:58.511341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.511309 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6"] Nov 01 22:03:58.512068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512051 5036 cpu_manager.go:414] "containerMap: removing container" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" Nov 01 22:03:58.512137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512069 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" Nov 01 22:03:58.512137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512094 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" containerName="revoke-keys" Nov 01 22:03:58.512137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512099 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" containerName="revoke-keys" Nov 01 22:03:58.512243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512210 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c5c171-9539-4e97-994e-23751c85579f" containerName="istio-proxy" Nov 01 22:03:58.512243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.512221 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" containerName="revoke-keys" Nov 01 22:03:58.517280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.517263 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.521505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.521288 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:03:58.522266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.522235 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:03:58.523620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.523600 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:03:58.523727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.523714 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:03:58.526069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.526045 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6"] Nov 01 22:03:58.566918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.566893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.567101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.567076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4zr\" (UniqueName: \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.567250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.567230 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.668792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.668754 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.668987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.668916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4zr\" (UniqueName: \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.669062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.669016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.669414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.669393 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.671371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.671353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.683403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.683378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4zr\" (UniqueName: \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\") pod \"maas-api-revoke-sub-7fb9c4de-5qzs6\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:58.744474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.744410 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:58.836618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:58.836585 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:03:59.136810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:59.136779 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6"] Nov 01 22:03:59.276091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:59.276056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" event={"ID":"12e92779-2a96-4be8-b04b-75be8d50dd15","Type":"ContainerStarted","Data":"19902bbaba22320f120cda871a85e7dd0f084ebeaff49e8aeeaff00251ffbfbf"} Nov 01 22:03:59.465357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:03:59.465261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:00.141482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:00.141416 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:00.292426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:00.292388 5036 generic.go:356] "Generic (PLEG): container finished" podID="12e92779-2a96-4be8-b04b-75be8d50dd15" containerID="f616d092845004ec7f1d4a0afe6dc115d0caacb16810edb1924552c596209965" exitCode=0 Nov 01 22:04:00.292669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:00.292515 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" event={"ID":"12e92779-2a96-4be8-b04b-75be8d50dd15","Type":"ContainerDied","Data":"f616d092845004ec7f1d4a0afe6dc115d0caacb16810edb1924552c596209965"} Nov 01 22:04:00.913363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:00.913326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:01.033239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.033209 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4"] Nov 01 22:04:01.040622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.040591 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j"] Nov 01 22:04:01.045358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.045333 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42189ac1-r5n6j"] Nov 01 22:04:01.049480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.049457 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0139ed-knds4"] Nov 01 22:04:01.054306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.054282 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:01.189607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.189517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:01.411706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.408582 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:04:01.436775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.436344 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.454990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.442360 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:04:01.455279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.444837 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edd32a-openshift-default-dockercfg-lkssj\"" Nov 01 22:04:01.481954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.480182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:01.529760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.526692 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.529760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.526811 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.529760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.526843 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.529760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.526953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.530713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.527008 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.530713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.530261 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.530713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.530432 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgkr\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.530713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.530500 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.530713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.530539 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.567173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.567146 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:04:01.615298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.615236 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:01.632147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632100 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\") pod \"12e92779-2a96-4be8-b04b-75be8d50dd15\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " Nov 01 22:04:01.632309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632293 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\" (UniqueName: \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\") pod \"12e92779-2a96-4be8-b04b-75be8d50dd15\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " Nov 01 22:04:01.632548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632527 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\") pod \"12e92779-2a96-4be8-b04b-75be8d50dd15\" (UID: \"12e92779-2a96-4be8-b04b-75be8d50dd15\") " Nov 01 22:04:01.632894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkr\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.632988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632920 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.632988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.632958 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633081 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca" pod "12e92779-2a96-4be8-b04b-75be8d50dd15" (UID: "12e92779-2a96-4be8-b04b-75be8d50dd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:01.633227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633161 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633191 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633304 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633359 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633397 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.633610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.633591 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:01.634940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.634912 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.635211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.635187 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.635480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.635459 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.635583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.635481 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.636251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.636225 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.639053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.639026 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.639280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.639259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.640141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.640112 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr" pod "12e92779-2a96-4be8-b04b-75be8d50dd15" (UID: "12e92779-2a96-4be8-b04b-75be8d50dd15"). InnerVolumeSpecName "kube-api-access-vc4zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:01.640567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.640542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca" pod "12e92779-2a96-4be8-b04b-75be8d50dd15" (UID: "12e92779-2a96-4be8-b04b-75be8d50dd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:01.646945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.646919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.648391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.648358 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgkr\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:01.854513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.847238 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4zr\" (UniqueName: \"kubernetes.io/projected/12e92779-2a96-4be8-b04b-75be8d50dd15-kube-api-access-vc4zr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:01.854513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.847288 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e92779-2a96-4be8-b04b-75be8d50dd15-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:01.934469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:01.929764 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:02.135055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.134901 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:02.337620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.337586 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" event={"ID":"12e92779-2a96-4be8-b04b-75be8d50dd15","Type":"ContainerDied","Data":"19902bbaba22320f120cda871a85e7dd0f084ebeaff49e8aeeaff00251ffbfbf"} Nov 01 22:04:02.337620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.337608 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6" Nov 01 22:04:02.338160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.337672 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19902bbaba22320f120cda871a85e7dd0f084ebeaff49e8aeeaff00251ffbfbf" Nov 01 22:04:02.441331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.441250 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b67b27-c12c-49ae-aa4c-b043da0c8e15" path="/var/lib/kubelet/pods/73b67b27-c12c-49ae-aa4c-b043da0c8e15/volumes" Nov 01 22:04:02.441768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.441751 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9feb9d45-8108-4a2f-af10-eaa3a10531ec" path="/var/lib/kubelet/pods/9feb9d45-8108-4a2f-af10-eaa3a10531ec/volumes" Nov 01 22:04:02.488193 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:04:02.488162 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1fb377_5ded_45dc_a75f_f98b4a3caf70.slice/crio-14ae4d926db23ad24a60a84159f33c24ad8a2eb1c13d5de97dde8baff8ce2e15 WatchSource:0}: Error finding container 14ae4d926db23ad24a60a84159f33c24ad8a2eb1c13d5de97dde8baff8ce2e15: Status 404 returned error can't find the container with id 14ae4d926db23ad24a60a84159f33c24ad8a2eb1c13d5de97dde8baff8ce2e15 Nov 01 22:04:02.491954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.491927 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:04:02.492073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.492035 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:02.492149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.492124 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:02.492208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.492170 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:02.937671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:02.937631 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:03.363586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.363549 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerStarted","Data":"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2"} Nov 01 22:04:03.364084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.363612 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerStarted","Data":"14ae4d926db23ad24a60a84159f33c24ad8a2eb1c13d5de97dde8baff8ce2e15"} Nov 01 22:04:03.394429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.394375 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podStartSLOduration=2.394360618 podStartE2EDuration="2.394360618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:03.3912026 +0000 UTC m=+1197.084183087" watchObservedRunningTime="2026-11-01 22:04:03.394360618 +0000 UTC m=+1197.087341103" Nov 01 22:04:03.929461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.929418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:03.941050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.941015 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:03.971021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:03.970977 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:04.034190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.034147 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv"] Nov 01 22:04:04.042043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.042008 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0a9228-wblgv"] Nov 01 22:04:04.372715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.372686 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:04.373776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.373751 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:04:04.453414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.453364 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35d1417-27c7-499b-8212-107af3192ec1" path="/var/lib/kubelet/pods/a35d1417-27c7-499b-8212-107af3192ec1/volumes" Nov 01 22:04:04.922753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:04.922718 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:05.750128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:05.750096 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:05.887519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:05.887477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:06.093544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:06.093504 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:06.418250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:06.409034 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:07.010471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:07.009768 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:04:07.121468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:07.120487 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:04:07.702886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:07.702080 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:08.065516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.056278 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9"] Nov 01 22:04:08.065516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.061784 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe5d4f5-5f7n9"] Nov 01 22:04:08.199472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.171625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:08.359529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.353436 5036 scope.go:122] "RemoveContainer" containerID="4d7d6a6abe18c375e0173e4a87356cbadb73ac6131b18a089acba2bd9906a6fe" Nov 01 22:04:08.508897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.508796 5036 scope.go:122] "RemoveContainer" containerID="28eef33569ba437c5a59ddd043312d5d442855ea6ddebd907af8c6771ce9044e" Nov 01 22:04:08.515318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.515279 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd67b024-7667-4c62-9c17-aeb82144c8b7" path="/var/lib/kubelet/pods/cd67b024-7667-4c62-9c17-aeb82144c8b7/volumes" Nov 01 22:04:08.533434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.533405 5036 scope.go:122] "RemoveContainer" containerID="dacce5b33d66817ecf0a49f86de0efa9a8a4b291b7642c156ab03c8550a9cb55" Nov 01 22:04:08.545881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.545860 5036 scope.go:122] "RemoveContainer" containerID="c822f41f19b1febb68c7e62630893c5e1eb9386e9df088c00b15d95f87604c8f" Nov 01 22:04:08.557602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.557584 5036 scope.go:122] "RemoveContainer" containerID="4d2c654a39dd9329cdeed2decd073644187440acf87b3444196d5f179d4b26dc" Nov 01 22:04:08.581151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.581127 5036 scope.go:122] "RemoveContainer" containerID="011b209a2d444ed822a96bd39891eba30e6b7562c3dd74a966956a40b9c870dc" Nov 01 22:04:08.601496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.600659 5036 scope.go:122] "RemoveContainer" containerID="b3fae844aa3c10604e7d493d72c92f0b5295e5c38b70792bc5d71aa27e12748a" Nov 01 22:04:08.630744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.624051 5036 scope.go:122] "RemoveContainer" containerID="57f2333358f40c92c67d7fb9e4fd2e483e2059be8b0fdb6bef64e89e571644f4" Nov 01 22:04:08.644881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.644861 5036 scope.go:122] "RemoveContainer" containerID="5a729ba27b1143acc242b320252ad5d7bb55b3d3a7cb77643b30cc1ab21eaa2a" Nov 01 22:04:08.656033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:08.656011 5036 scope.go:122] "RemoveContainer" containerID="255e394d0c87acb0223cbd2567d0bbb528ff6de0419786d98a67113c02455327" Nov 01 22:04:09.109035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:09.109001 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:09.825270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:09.825231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:10.787955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:10.787918 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:11.205617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:11.205521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:11.716659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:11.716616 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:11.844278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:11.844229 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:12.028774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.028744 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7"] Nov 01 22:04:12.033110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.033086 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972d2493-ptph7"] Nov 01 22:04:12.186873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.186843 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:04:12.188154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.188125 5036 cpu_manager.go:414] "containerMap: removing container" podUID="12e92779-2a96-4be8-b04b-75be8d50dd15" containerName="revoke-keys" Nov 01 22:04:12.188343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.188330 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e92779-2a96-4be8-b04b-75be8d50dd15" containerName="revoke-keys" Nov 01 22:04:12.188770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.188753 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e92779-2a96-4be8-b04b-75be8d50dd15" containerName="revoke-keys" Nov 01 22:04:12.200534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.200511 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:04:12.200851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.200770 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.215137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.214499 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-004180-openshift-default-dockercfg-9v46l\"" Nov 01 22:04:12.248318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.248266 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:12.328529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.317499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.334571 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.334651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.334683 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.334839 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.334963 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbgr\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.335044 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.335142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.340470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.335271 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.511336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.511290 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.513625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.513381 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.514156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.514133 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.519463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.513798 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.519792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.519772 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.520272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.520254 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.523769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.523752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.524414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.524396 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.525012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.523690 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.525148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.524842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.525247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.520192 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.525458 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.525427 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbgr\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.526038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.526019 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.527470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.526574 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.537484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.534252 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.537617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.537538 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42ea635-8050-459b-bf1c-ba629317bfb0" path="/var/lib/kubelet/pods/c42ea635-8050-459b-bf1c-ba629317bfb0/volumes" Nov 01 22:04:12.541747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.541724 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.572466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.569948 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.591515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.588159 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbgr\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\") pod \"e2e-api-b-004180-openshift-default-7dfb78476f-25d2k\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.691769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.691573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:12.852223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.852135 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:12.975722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:12.963367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:13.158659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.158629 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:04:13.159013 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:04:13.158989 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d056d7_ba24_40f6_8979_fe61cb7b7da8.slice/crio-0eab5a4d4517bb90602c284df4b5938d0dc329a9356d8f2781655ac78c09065f WatchSource:0}: Error finding container 0eab5a4d4517bb90602c284df4b5938d0dc329a9356d8f2781655ac78c09065f: Status 404 returned error can't find the container with id 0eab5a4d4517bb90602c284df4b5938d0dc329a9356d8f2781655ac78c09065f Nov 01 22:04:13.161852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.161817 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:13.161947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.161879 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:13.161947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.161906 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:13.472556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.472188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:13.808469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.805295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerStarted","Data":"650ae3df6ea3cfeca4dad308e68a67d5fe43bfecd75cf543c9b8c64695813132"} Nov 01 22:04:13.808469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.805369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerStarted","Data":"0eab5a4d4517bb90602c284df4b5938d0dc329a9356d8f2781655ac78c09065f"} Nov 01 22:04:13.831761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.831705 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podStartSLOduration=1.831691511 podStartE2EDuration="1.831691511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:13.829251263 +0000 UTC m=+1207.522231762" watchObservedRunningTime="2026-11-01 22:04:13.831691511 +0000 UTC m=+1207.524671997" Nov 01 22:04:13.852841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:13.852814 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:14.455401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:14.455365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:14.858629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:14.858600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:15.024769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:15.024735 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk"] Nov 01 22:04:15.029626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:15.029597 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135be121-cjlmk"] Nov 01 22:04:15.299462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:15.299273 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:15.827115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:15.827084 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:15.828232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:15.828208 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:04:16.047871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:16.047830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:16.177431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:16.177342 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:16.320373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:16.320338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:16.456499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:16.456359 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1481809f-ee9d-4668-8b43-09833840840c" path="/var/lib/kubelet/pods/1481809f-ee9d-4668-8b43-09833840840c/volumes" Nov 01 22:04:17.248770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:17.248736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:17.438608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:17.403907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:18.510030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:18.509999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:18.861650 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:04:18.861614 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:04:19.591798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:19.591745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:20.275827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:20.275792 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:20.528532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:20.528380 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:20.600374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:20.599206 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj"] Nov 01 22:04:20.680836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:20.677066 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-vzkkj"] Nov 01 22:04:20.680836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:20.678349 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:21.354002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:21.353965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:21.700304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:21.700178 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:21.830661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:21.830627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:22.045612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.045542 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd"] Nov 01 22:04:22.107529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.107498 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956642ac-f5kwd"] Nov 01 22:04:22.443836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.443747 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:22.452144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.452110 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0635d7c5-1911-4977-801b-78a52c4ae31a" path="/var/lib/kubelet/pods/0635d7c5-1911-4977-801b-78a52c4ae31a/volumes" Nov 01 22:04:22.452940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.452905 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e75bfbb-6c3a-4132-9773-24a6257f5795" path="/var/lib/kubelet/pods/2e75bfbb-6c3a-4132-9773-24a6257f5795/volumes" Nov 01 22:04:22.568202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.568166 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:22.700314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:22.700208 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:23.108468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:23.107490 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd"] Nov 01 22:04:23.138471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:23.130215 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0946ec3d-kgfrd"] Nov 01 22:04:23.552811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:23.552773 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:24.406936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:24.406898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:24.454108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:24.454072 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558aea62-4f7c-47bd-b406-3c8a59a2769b" path="/var/lib/kubelet/pods/558aea62-4f7c-47bd-b406-3c8a59a2769b/volumes" Nov 01 22:04:24.539601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:24.539555 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:25.245100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:25.245053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:25.403280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:25.403243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:25.535393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:25.535365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:26.358366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:26.357019 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:26.683549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:26.683467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:26.936516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:26.935824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:27.446650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:27.446612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:27.579665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:27.579620 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:27.778332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:27.778297 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:28.033016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:28.032928 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:28.206467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:28.200095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:29.146265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:29.146229 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:30.323964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:30.323931 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:31.134274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:31.134242 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:31.801803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:31.801764 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:32.459274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:32.459235 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:32.587836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:32.587807 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:32.739655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:32.739559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:33.426257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:33.426207 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:33.703277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:33.703185 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 22:04:33.703440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:33.703404 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" containerID="cri-o://f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1" gracePeriod=30 Nov 01 22:04:34.363188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:34.363148 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:35.133277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:35.133242 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:35.826103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:35.826060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:36.545967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:36.545876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:37.238585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:37.238534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:37.382194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:37.382131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:37.514670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:37.514584 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:38.229715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:38.229675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:39.051435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.051410 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 22:04:39.092468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.092422 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:39.120004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.119969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.120172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120022 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.120244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120219 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.120305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120291 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.120369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120306 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:39.120369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.120520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120384 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120830 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120892 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.120965 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\") pod \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\" (UID: \"90a0abae-96d9-4d9a-beaa-97e6d61922e8\") " Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.121066 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.121148 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:39.122538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.121741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:39.123532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.123430 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.123532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.123468 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.123532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.123485 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.123532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.123499 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.123532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.123512 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.125148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.125122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "kube-api-access-68nqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:39.128918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.125564 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:39.128918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.125576 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:04:39.128918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.127505 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token" pod "90a0abae-96d9-4d9a-beaa-97e6d61922e8" (UID: "90a0abae-96d9-4d9a-beaa-97e6d61922e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:39.212252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.212209 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:39.225726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.225693 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.225726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.225727 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nqx\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-kube-api-access-68nqx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.225924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.225743 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.225924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.225757 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a0abae-96d9-4d9a-beaa-97e6d61922e8-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:39.438794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.438761 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:04:39.439894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.439870 5036 cpu_manager.go:414] "containerMap: removing container" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" Nov 01 22:04:39.439991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.439897 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" Nov 01 22:04:39.440101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.440087 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerName="istio-proxy" Nov 01 22:04:39.444829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.444804 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.461700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.461676 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-702002-openshift-default-dockercfg-2lbqw\"" Nov 01 22:04:39.471471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.468144 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:04:39.519828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.519781 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:39.531684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.531651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.532195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.532176 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.613705 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.613762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstzw\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.613798 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.613846 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.614007 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.614114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.636502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.614204 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.726544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.726054 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" Nov 01 22:04:39.726728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.726556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" event={"ID":"90a0abae-96d9-4d9a-beaa-97e6d61922e8","Type":"ContainerDied","Data":"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1"} Nov 01 22:04:39.726728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.726636 5036 scope.go:122] "RemoveContainer" containerID="f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1" Nov 01 22:04:39.726871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.725927 5036 generic.go:356] "Generic (PLEG): container finished" podID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" containerID="f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1" exitCode=0 Nov 01 22:04:39.738546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.738490 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv" event={"ID":"90a0abae-96d9-4d9a-beaa-97e6d61922e8","Type":"ContainerDied","Data":"39721ce3db145a59f2d841ccbf2abdfcb1d9975ed3636aab3531876a15769604"} Nov 01 22:04:39.771484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.764877 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.781518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.773703 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.781518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.774022 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstzw\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.781518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.774156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.781518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.774301 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.781518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.774383 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.782468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.782398 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.782612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.782558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.782679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.782662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.787493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.782742 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.787493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.782781 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.809502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.808257 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.809502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.809399 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.809979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.809934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:39.899493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:39.899458 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstzw\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:40.076254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.062046 5036 scope.go:122] "RemoveContainer" containerID="f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1" Nov 01 22:04:40.077100 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:04:40.076972 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1\": container with ID starting with f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1 not found: ID does not exist" containerID="f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1" Nov 01 22:04:40.077100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.077020 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1"} err="failed to get container status \"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1\": rpc error: code = NotFound desc = could not find container \"f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1\": container with ID starting with f3b088347d1e5b8457d7c91df96d96453550310223ae8d8f7de1bdb47880c0e1 not found: ID does not exist" Nov 01 22:04:40.133708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.133633 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:40.148476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.139513 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:40.149022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.148959 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:40.245465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.226978 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 22:04:40.289504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.284256 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-fbrkv"] Nov 01 22:04:40.335556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.335429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:40.393641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.393323 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:40.483497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.483460 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a0abae-96d9-4d9a-beaa-97e6d61922e8" path="/var/lib/kubelet/pods/90a0abae-96d9-4d9a-beaa-97e6d61922e8/volumes" Nov 01 22:04:40.508508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.508463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:40.807859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.807400 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:04:40.822148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.821573 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:40.822148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.821663 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:40.822148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:40.821706 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:41.110512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.109702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.154842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.151056 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd"] Nov 01 22:04:41.157755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.157612 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.163063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.162978 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:41.164421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.163491 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:04:41.164421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.163570 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:41.164421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.163602 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:41.167016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.166674 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd"] Nov 01 22:04:41.186357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.183280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmt7\" (UniqueName: \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.186357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.183353 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.186357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.183387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.284930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.284889 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmt7\" (UniqueName: \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.285135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.285014 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.285135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.285074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.286271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.286242 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.301324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.301120 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmt7\" (UniqueName: \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.306199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.306173 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.490370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.490284 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:41.519702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.519601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.653041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.653007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.803103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.802842 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.832561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.831401 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerStarted","Data":"d661453ba9a870dc2e9dd93b3293b9e60a244afd9eb8b3b272b74979f8bcb578"} Nov 01 22:04:41.832561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.831495 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerStarted","Data":"759aa5d3718ac9d5320d952930bbf4f01964e309bbd3771f06901bc9d8636c9d"} Nov 01 22:04:41.833418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.833389 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd"] Nov 01 22:04:41.841239 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:04:41.841214 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6ab562_54c7_4add_91ad_e667a303e633.slice/crio-1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc WatchSource:0}: Error finding container 1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc: Status 404 returned error can't find the container with id 1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc Nov 01 22:04:41.863489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.863270 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podStartSLOduration=2.863251655 podStartE2EDuration="2.863251655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:41.85842671 +0000 UTC m=+1235.551407196" watchObservedRunningTime="2026-11-01 22:04:41.863251655 +0000 UTC m=+1235.556232142" Nov 01 22:04:41.960532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.958501 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:04:41.973158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.973127 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.982406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.982383 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:41.985615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.982275 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:04:41.988976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:41.988522 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3577a0-openshift-default-dockercfg-nmmqq\"" Nov 01 22:04:42.007591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.007568 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:04:42.016270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016238 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016284 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcx4\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016389 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016519 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016560 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016614 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016764 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.016862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.016845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.107463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.107364 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:42.119074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119028 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcx4\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119260 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119297 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119669 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.119846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.119730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.120261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.120236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.120418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.120341 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.120836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.120815 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.121114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.121086 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.121414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.121389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.123764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.123742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.130796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.130768 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.133277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.133246 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.133376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.133343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcx4\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.322469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.315047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:42.322469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.315628 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:42.403276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.403189 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:42.773032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.772672 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:42.858490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.856086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" event={"ID":"2b6ab562-54c7-4add-91ad-e667a303e633","Type":"ContainerStarted","Data":"72f07ccc1b0c09425c857f509f8152e244e35faa91d0e46984566f1d826a54ab"} Nov 01 22:04:42.858490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.856145 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" event={"ID":"2b6ab562-54c7-4add-91ad-e667a303e633","Type":"ContainerStarted","Data":"1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc"} Nov 01 22:04:42.858490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.856173 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:42.858490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.857919 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:04:42.890494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:42.883177 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" podStartSLOduration=1.883159137 podStartE2EDuration="1.883159137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:42.875840229 +0000 UTC m=+1236.568820713" watchObservedRunningTime="2026-11-01 22:04:42.883159137 +0000 UTC m=+1236.576139624" Nov 01 22:04:43.027953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.027921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:43.039939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.039909 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:04:43.071875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.056266 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:43.071875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.056365 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:43.071875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.056412 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:04:43.876983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.876944 5036 generic.go:356] "Generic (PLEG): container finished" podID="2b6ab562-54c7-4add-91ad-e667a303e633" containerID="72f07ccc1b0c09425c857f509f8152e244e35faa91d0e46984566f1d826a54ab" exitCode=0 Nov 01 22:04:43.879052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.878393 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" event={"ID":"2b6ab562-54c7-4add-91ad-e667a303e633","Type":"ContainerDied","Data":"72f07ccc1b0c09425c857f509f8152e244e35faa91d0e46984566f1d826a54ab"} Nov 01 22:04:43.884999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.883712 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerStarted","Data":"7e2eb5fc5e0891f6722c72c15186e591c25d7645b76228f3bd89e5a9ca19b24f"} Nov 01 22:04:43.884999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.883766 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerStarted","Data":"ad873a66ee03fe9a64cae664b4b22fe77f8664f6a529dadd277c088abf67384b"} Nov 01 22:04:43.937484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:43.929189 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podStartSLOduration=2.929171251 podStartE2EDuration="2.929171251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:43.927166557 +0000 UTC m=+1237.620147043" watchObservedRunningTime="2026-11-01 22:04:43.929171251 +0000 UTC m=+1237.622151741" Nov 01 22:04:44.321466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.320729 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:44.321466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.320846 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:04:44.321466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.320923 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:04:44.730684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.725307 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:44.744856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.744809 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:44.799595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.799554 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:44.874365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.874322 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:44.925048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:44.925009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:45.023527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.002514 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.071464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.068565 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.308556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.296543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.308556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.307638 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.330469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.321850 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:04:45.330469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.321908 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:04:45.804115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.803807 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:45.889103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.889013 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:45.965179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.951211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\" (UniqueName: \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\") pod \"2b6ab562-54c7-4add-91ad-e667a303e633\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " Nov 01 22:04:45.965179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.951271 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\") pod \"2b6ab562-54c7-4add-91ad-e667a303e633\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " Nov 01 22:04:45.965179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.951418 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\") pod \"2b6ab562-54c7-4add-91ad-e667a303e633\" (UID: \"2b6ab562-54c7-4add-91ad-e667a303e633\") " Nov 01 22:04:45.965179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.957986 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca" pod "2b6ab562-54c7-4add-91ad-e667a303e633" (UID: "2b6ab562-54c7-4add-91ad-e667a303e633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:45.966699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.966662 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:45.975258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:45.975137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca" pod "2b6ab562-54c7-4add-91ad-e667a303e633" (UID: "2b6ab562-54c7-4add-91ad-e667a303e633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:46.013081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.013016 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7" pod "2b6ab562-54c7-4add-91ad-e667a303e633" (UID: "2b6ab562-54c7-4add-91ad-e667a303e633"). InnerVolumeSpecName "kube-api-access-ptmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:46.016696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.016667 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" Nov 01 22:04:46.027958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.020612 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd" event={"ID":"2b6ab562-54c7-4add-91ad-e667a303e633","Type":"ContainerDied","Data":"1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc"} Nov 01 22:04:46.027958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.020666 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb9fc803f0b707c232635e3243e3ac3d5bdac8a4b9ecaf563834a8b0ee5d1cc" Nov 01 22:04:46.072467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.071609 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmt7\" (UniqueName: \"kubernetes.io/projected/2b6ab562-54c7-4add-91ad-e667a303e633-kube-api-access-ptmt7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:46.072467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.071640 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6ab562-54c7-4add-91ad-e667a303e633-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:46.326736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:46.326703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:47.129465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.128760 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:47.598957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.598921 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:47.637755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.637304 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:04:47.689099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.688549 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:47.741468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.724801 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:47.750745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:47.750700 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.109082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:48.108981 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.140423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:48.140300 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:04:48.141515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:48.141412 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.310460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:48.309578 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:04:48.889464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:48.888912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:49.026369 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:04:49.026328 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:04:49.827517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:49.827389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:49.969118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:49.969082 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:50.957529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:50.957489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:51.094047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:51.094007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:51.570067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:51.570031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:51.706692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:51.706658 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:52.403600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:52.403562 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:53.239272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:53.239221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:54.107146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:54.107105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:55.037001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:55.036961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:55.749437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:55.749390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:56.760797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:56.760764 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:57.416254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:57.416200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:57.631894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:57.631855 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:58.319091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:58.319059 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:58.973110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:58.973016 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:59.129155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.129126 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:59.306064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.306027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:59.564688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.564591 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd"] Nov 01 22:04:59.572625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.572585 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b1b5598d-5ljxd"] Nov 01 22:04:59.641722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.641689 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:59.850866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:04:59.850776 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:00.440350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:00.440321 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6ab562-54c7-4add-91ad-e667a303e633" path="/var/lib/kubelet/pods/2b6ab562-54c7-4add-91ad-e667a303e633/volumes" Nov 01 22:05:00.546559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:00.546516 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:01.716701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:01.716658 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:01.847699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:01.847660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:02.944346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:02.944312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:03.857596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:03.857556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:04.038117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.038077 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp"] Nov 01 22:05:04.039915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.039887 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2b6ab562-54c7-4add-91ad-e667a303e633" containerName="revoke-keys" Nov 01 22:05:04.040116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.040090 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6ab562-54c7-4add-91ad-e667a303e633" containerName="revoke-keys" Nov 01 22:05:04.040494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.040474 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6ab562-54c7-4add-91ad-e667a303e633" containerName="revoke-keys" Nov 01 22:05:04.046711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.046688 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.055116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.055092 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:05:04.057412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.057389 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:04.057846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.057658 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:04.057982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.057737 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:04.066391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.066355 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp"] Nov 01 22:05:04.092052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.092018 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtlf\" (UniqueName: \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.092206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.092148 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.092361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.092342 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.193715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.193616 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.193901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.193734 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtlf\" (UniqueName: \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.193901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.193846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.194824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.194793 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.196178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.196150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.206616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.206593 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtlf\" (UniqueName: \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\") pod \"maas-api-revoke-keys-e2e-api-a-d506c6-z98tp\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.384460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.384419 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:04.682175 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:05:04.682150 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506c1455_71ab_42d7_9740_f45b771dd6a7.slice/crio-3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18 WatchSource:0}: Error finding container 3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18: Status 404 returned error can't find the container with id 3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18 Nov 01 22:05:04.683852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.683823 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp"] Nov 01 22:05:04.812161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:04.812120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:05.018317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.018283 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" event={"ID":"506c1455-71ab-42d7-9740-f45b771dd6a7","Type":"ContainerStarted","Data":"aecb916e12a9b0adc3fe221582b52163bc667297928d73d6be3d7aa306c57c54"} Nov 01 22:05:05.018469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.018340 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" event={"ID":"506c1455-71ab-42d7-9740-f45b771dd6a7","Type":"ContainerStarted","Data":"3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18"} Nov 01 22:05:05.038021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.037971 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" podStartSLOduration=1.037959496 podStartE2EDuration="1.037959496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:05.037321315 +0000 UTC m=+1258.730301805" watchObservedRunningTime="2026-11-01 22:05:05.037959496 +0000 UTC m=+1258.730939999" Nov 01 22:05:05.523800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.523765 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6956-a75e13"] Nov 01 22:05:05.528582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.528558 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:05.534838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.534795 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 22:05:05.536155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.536133 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6956-a75e13"] Nov 01 22:05:05.616993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.616961 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhx4m\" (UniqueName: \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\") pod \"test-curl-6956-a75e13\" (UID: \"e66a7573-7cb2-4869-9031-2e64172c16ef\") " pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:05.719102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.719074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhx4m\" (UniqueName: \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\") pod \"test-curl-6956-a75e13\" (UID: \"e66a7573-7cb2-4869-9031-2e64172c16ef\") " pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:05.731509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.731485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhx4m\" (UniqueName: \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\") pod \"test-curl-6956-a75e13\" (UID: \"e66a7573-7cb2-4869-9031-2e64172c16ef\") " pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:05.854859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.854761 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:05.988482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:05.979123 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:06.097414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.095687 5036 generic.go:356] "Generic (PLEG): container finished" podID="506c1455-71ab-42d7-9740-f45b771dd6a7" containerID="aecb916e12a9b0adc3fe221582b52163bc667297928d73d6be3d7aa306c57c54" exitCode=0 Nov 01 22:05:06.097414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.096045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" event={"ID":"506c1455-71ab-42d7-9740-f45b771dd6a7","Type":"ContainerDied","Data":"aecb916e12a9b0adc3fe221582b52163bc667297928d73d6be3d7aa306c57c54"} Nov 01 22:05:06.373255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.373227 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6956-a75e13"] Nov 01 22:05:06.397807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.397782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:06.533278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.533245 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:06.660687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.660578 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:05:06.660918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.660779 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" containerID="cri-o://e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70" gracePeriod=30 Nov 01 22:05:06.729333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:06.729288 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:07.106013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.105975 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6956-a75e13" event={"ID":"e66a7573-7cb2-4869-9031-2e64172c16ef","Type":"ContainerStarted","Data":"51958886e319ff1ada9f91a306275dcec533b9716979a8eebde7a0e994daee66"} Nov 01 22:05:07.106211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.106026 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6956-a75e13" event={"ID":"e66a7573-7cb2-4869-9031-2e64172c16ef","Type":"ContainerStarted","Data":"5df77c1c27264c92d6a0a6210803dd5908067e80f0593e2ff593c5f0a44d5f58"} Nov 01 22:05:07.123313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.123246 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6956-a75e13" podStartSLOduration=2.123227417 podStartE2EDuration="2.123227417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:07.122829349 +0000 UTC m=+1260.815809836" watchObservedRunningTime="2026-11-01 22:05:07.123227417 +0000 UTC m=+1260.816207930" Nov 01 22:05:07.321519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.321492 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:07.331959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.331923 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:07.396774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.396687 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\") pod \"506c1455-71ab-42d7-9740-f45b771dd6a7\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " Nov 01 22:05:07.397069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.397050 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\" (UniqueName: \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\") pod \"506c1455-71ab-42d7-9740-f45b771dd6a7\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " Nov 01 22:05:07.397295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.397279 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\") pod \"506c1455-71ab-42d7-9740-f45b771dd6a7\" (UID: \"506c1455-71ab-42d7-9740-f45b771dd6a7\") " Nov 01 22:05:07.403180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.400829 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca" pod "506c1455-71ab-42d7-9740-f45b771dd6a7" (UID: "506c1455-71ab-42d7-9740-f45b771dd6a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:07.405670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.404399 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca" pod "506c1455-71ab-42d7-9740-f45b771dd6a7" (UID: "506c1455-71ab-42d7-9740-f45b771dd6a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:07.414852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.414820 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf" pod "506c1455-71ab-42d7-9740-f45b771dd6a7" (UID: "506c1455-71ab-42d7-9740-f45b771dd6a7"). InnerVolumeSpecName "kube-api-access-8mtlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:07.468628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.468591 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:07.501254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.501225 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:07.501254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.501251 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtlf\" (UniqueName: \"kubernetes.io/projected/506c1455-71ab-42d7-9740-f45b771dd6a7-kube-api-access-8mtlf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:07.501495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.501269 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506c1455-71ab-42d7-9740-f45b771dd6a7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:07.606154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.606114 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:07.879295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.879254 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:05:07.880390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.880362 5036 cpu_manager.go:414] "containerMap: removing container" podUID="506c1455-71ab-42d7-9740-f45b771dd6a7" containerName="revoke-keys" Nov 01 22:05:07.880390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.880390 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506c1455-71ab-42d7-9740-f45b771dd6a7" containerName="revoke-keys" Nov 01 22:05:07.880696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.880678 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="506c1455-71ab-42d7-9740-f45b771dd6a7" containerName="revoke-keys" Nov 01 22:05:07.892713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.892687 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:07.937285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.937149 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bb882c-openshift-default-dockercfg-pws8w\"" Nov 01 22:05:07.958660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.958488 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:05:07.976249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.976182 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6956-a75e13"] Nov 01 22:05:07.976433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:07.976258 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6956-a75e13"] Nov 01 22:05:08.063686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.051918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.063964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.063932 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.064103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.064086 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.064211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.064197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.064512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.064490 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.107464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.096731 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzs5\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.107464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.096801 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.107464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.096867 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.107464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.097025 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219606 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219723 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzs5\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219949 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.219985 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.220515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.220080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.287936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.287900 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.288982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.288945 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.296623 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.296891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.297127 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.312910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:08.571689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.372058 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.391363 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzs5\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.571689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.495081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.576711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.576687 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:08.577080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.577052 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" event={"ID":"506c1455-71ab-42d7-9740-f45b771dd6a7","Type":"ContainerDied","Data":"3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18"} Nov 01 22:05:08.594882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.577111 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e648a7cea842fe26175403b022312e255fd6eeaf4638db7b1aff841acf99f18" Nov 01 22:05:08.595003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.594897 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc"] Nov 01 22:05:08.596066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.595994 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e66a7573-7cb2-4869-9031-2e64172c16ef" containerName="test-curl-6956-a75e13" Nov 01 22:05:08.596245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.596226 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66a7573-7cb2-4869-9031-2e64172c16ef" containerName="test-curl-6956-a75e13" Nov 01 22:05:08.596628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.596605 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66a7573-7cb2-4869-9031-2e64172c16ef" containerName="test-curl-6956-a75e13" Nov 01 22:05:08.598995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.598961 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\") pod \"e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.604815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.604798 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc"] Nov 01 22:05:08.605069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.605017 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.606860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.606022 5036 patch_prober.go:29] interesting pod/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:05:08.607338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.607314 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:05:08.610789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.610762 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:05:08.612163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.612005 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:08.613569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.613364 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:08.613569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.613422 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:08.702317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.702280 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:08.723996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.723965 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:08.729529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.729496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsbg\" (UniqueName: \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.729690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.729619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.729900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.729876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.779258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.778257 5036 scope.go:122] "RemoveContainer" containerID="716519b3de85e19f64cdeff6916de03801d49e046e29cfde0504aaa0607cdb9e" Nov 01 22:05:08.791629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.791592 5036 scope.go:122] "RemoveContainer" containerID="485a7e4e33adbb9f82ddaf1bfe0992769e8276088eaad01b3e008c28d2694065" Nov 01 22:05:08.812226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.812199 5036 scope.go:122] "RemoveContainer" containerID="e30cebe9d24e404bd26bf03a795993b2c4b5015a44b721063ceed581834878d2" Nov 01 22:05:08.834524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.831489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.834524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.831709 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsbg\" (UniqueName: \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.834524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.831764 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.834524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.833012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:08.834524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.834097 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.840856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.834991 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.840856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.839584 5036 scope.go:122] "RemoveContainer" containerID="339c33faec560dc83c27fe0cc89e6c8a13782fd85a1964e499ae7f274d727cb4" Nov 01 22:05:08.920356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.913479 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsbg\" (UniqueName: \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\") pod \"maas-api-revoke-keys-e2e-mt-21dd16de-p77zc\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.920356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.917104 5036 scope.go:122] "RemoveContainer" containerID="d4732cf96f1b73e494bf71dd612af9a1c5f109ddaad6b8c34a6f716e91ca9901" Nov 01 22:05:08.935130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.935100 5036 scope.go:122] "RemoveContainer" containerID="bdb63bf046c568a094a77b72e339c99ba50034abc3f7335e201cba2de23993a0" Nov 01 22:05:08.944771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.940308 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:08.966647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.966616 5036 scope.go:122] "RemoveContainer" containerID="7bab04eb59523a7419fed49e81e7e27f5458a91a80b8d10e36289b6e93416a91" Nov 01 22:05:08.996297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:08.995928 5036 scope.go:122] "RemoveContainer" containerID="e83084dbd4fef08eb75866545e312bb9ec256341819f9952edecaa9cad845a0b" Nov 01 22:05:09.011633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.011261 5036 scope.go:122] "RemoveContainer" containerID="10502d7dd4293bb9c51f8c13690efd37c7001046056f854437dd9cfcbe66c1bf" Nov 01 22:05:09.029311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.029290 5036 scope.go:122] "RemoveContainer" containerID="65acfe8c4aa576a0c52e4c5b4726c8b26e9f10064c1be33889ea229171ad2a08" Nov 01 22:05:09.044297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.044266 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:05:09.049036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.049015 5036 scope.go:122] "RemoveContainer" containerID="ce4911021d4b82c7a3bc0a0e2578390a2f86289c412ae6601e6a7b817e49a644" Nov 01 22:05:09.053829 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:05:09.053797 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4c32b4_f484_4287_ba7b_ba56a80d1e55.slice/crio-5b127435359df47896aa4695f88b2fa08099c17e8c3d4c0c34128ef817b655b5 WatchSource:0}: Error finding container 5b127435359df47896aa4695f88b2fa08099c17e8c3d4c0c34128ef817b655b5: Status 404 returned error can't find the container with id 5b127435359df47896aa4695f88b2fa08099c17e8c3d4c0c34128ef817b655b5 Nov 01 22:05:09.058178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.057460 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:09.058178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.057524 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:09.058178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.057560 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:09.068213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.068193 5036 scope.go:122] "RemoveContainer" containerID="2998108e54f9515815a24b88c080823d3b8963f64181b9ca050ad2d56d05a682" Nov 01 22:05:09.179556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.173433 5036 scope.go:122] "RemoveContainer" containerID="62547ce60a16d8c6538eb0d20b2e21c9e131a23fb8e270e5ab9597fa89b54b9f" Nov 01 22:05:09.243357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.243318 5036 scope.go:122] "RemoveContainer" containerID="15624c63be1feddbcedceb8303e845f4a9c0fb98b4a76669a163cac1fa85de65" Nov 01 22:05:09.271753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.271728 5036 scope.go:122] "RemoveContainer" containerID="a472bb2e7ade8eebacb2510486f7a0ae3af928d7ffd07330b28b95ae268469e6" Nov 01 22:05:09.307361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.307332 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc"] Nov 01 22:05:09.340320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.340293 5036 scope.go:122] "RemoveContainer" containerID="59536e7e9e10a28c90008754519b3c44bd2a24ee805e7e54c298e407b9c2395e" Nov 01 22:05:09.390942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.390660 5036 scope.go:122] "RemoveContainer" containerID="8f2bf59994c8030ddb7ef9f67a2915a45e83d536a78b561f54c6ccf01d103f78" Nov 01 22:05:09.431616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.431590 5036 scope.go:122] "RemoveContainer" containerID="15db8c4b5cb3ddfbf5c9c008f388bcd8e3d6c783014d1a0659a0ae5e05c94ee5" Nov 01 22:05:09.620655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.620624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerStarted","Data":"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123"} Nov 01 22:05:09.621251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.620677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerStarted","Data":"5b127435359df47896aa4695f88b2fa08099c17e8c3d4c0c34128ef817b655b5"} Nov 01 22:05:09.623651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.623618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" event={"ID":"3aaa9694-d751-4f13-bd84-6a6d714d3a7e","Type":"ContainerStarted","Data":"7e6d9c0400902fbc214f6a1200444d29abaf2f22d152d09c243b7d39c2c491be"} Nov 01 22:05:09.627646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.627304 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6956-a75e13" podUID="e66a7573-7cb2-4869-9031-2e64172c16ef" containerName="test-curl-6956-a75e13" containerID="cri-o://51958886e319ff1ada9f91a306275dcec533b9716979a8eebde7a0e994daee66" gracePeriod=2 Nov 01 22:05:09.649489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.646946 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podStartSLOduration=2.646930599 podStartE2EDuration="2.646930599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:09.644435299 +0000 UTC m=+1263.337415786" watchObservedRunningTime="2026-11-01 22:05:09.646930599 +0000 UTC m=+1263.339911084" Nov 01 22:05:09.724663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:09.724578 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:10.042213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:10.042176 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:10.173215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:10.171980 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:10.698468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:10.695109 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aaa9694-d751-4f13-bd84-6a6d714d3a7e" containerID="399708a88c5b3d48d26081eed1603290cb3657de76d349fd07858e5a54885b35" exitCode=0 Nov 01 22:05:10.698468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:10.696794 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" event={"ID":"3aaa9694-d751-4f13-bd84-6a6d714d3a7e","Type":"ContainerDied","Data":"399708a88c5b3d48d26081eed1603290cb3657de76d349fd07858e5a54885b35"} Nov 01 22:05:10.782115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:10.781772 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:11.115792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.115711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:11.243631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.243020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:11.724641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.724585 5036 generic.go:356] "Generic (PLEG): container finished" podID="e66a7573-7cb2-4869-9031-2e64172c16ef" containerID="51958886e319ff1ada9f91a306275dcec533b9716979a8eebde7a0e994daee66" exitCode=137 Nov 01 22:05:11.725540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.725511 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:11.726992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.726967 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:05:11.886216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.886121 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5"] Nov 01 22:05:11.918267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.918240 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:11.918925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.918137 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5"] Nov 01 22:05:11.924867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.924842 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:11.925215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.925182 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfdr\" (UniqueName: \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:11.925323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.925251 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:11.958847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.957411 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:11.991947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:11.991923 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:12.029818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.020871 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:12.030840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.030808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\") pod \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " Nov 01 22:05:12.030945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.030908 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\") pod \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " Nov 01 22:05:12.031176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.031161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\" (UniqueName: \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\") pod \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\" (UID: \"3aaa9694-d751-4f13-bd84-6a6d714d3a7e\") " Nov 01 22:05:12.031240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.031214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\" (UniqueName: \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\") pod \"e66a7573-7cb2-4869-9031-2e64172c16ef\" (UID: \"e66a7573-7cb2-4869-9031-2e64172c16ef\") " Nov 01 22:05:12.031668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.031644 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.031780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.031758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfdr\" (UniqueName: \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.031852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.031816 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.032831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.032805 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.033430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.033406 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca" pod "3aaa9694-d751-4f13-bd84-6a6d714d3a7e" (UID: "3aaa9694-d751-4f13-bd84-6a6d714d3a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:12.035817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.035656 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca" pod "3aaa9694-d751-4f13-bd84-6a6d714d3a7e" (UID: "3aaa9694-d751-4f13-bd84-6a6d714d3a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:12.042410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.042346 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.042729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.042684 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m" pod "e66a7573-7cb2-4869-9031-2e64172c16ef" (UID: "e66a7573-7cb2-4869-9031-2e64172c16ef"). InnerVolumeSpecName "kube-api-access-zhx4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:12.044741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.044719 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg" pod "3aaa9694-d751-4f13-bd84-6a6d714d3a7e" (UID: "3aaa9694-d751-4f13-bd84-6a6d714d3a7e"). InnerVolumeSpecName "kube-api-access-rpsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:12.056098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.056065 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfdr\" (UniqueName: \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\") pod \"maas-api-revoke-keys-e2e-api-a-a19582-tmhp5\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.100994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.100964 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:05:12.133590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.133556 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.133843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.133814 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.133978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.133952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134077 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134105 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134346 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.134539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.134399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\") pod \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\" (UID: \"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf\") " Nov 01 22:05:12.135035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135015 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.135129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135040 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsbg\" (UniqueName: \"kubernetes.io/projected/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-kube-api-access-rpsbg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.135129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135059 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhx4m\" (UniqueName: \"kubernetes.io/projected/e66a7573-7cb2-4869-9031-2e64172c16ef-kube-api-access-zhx4m\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.135129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135073 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9694-d751-4f13-bd84-6a6d714d3a7e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.135478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:12.135873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.135850 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:12.136147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.136120 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:12.136802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.136729 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:12.138632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.138603 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:12.140651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.140631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "kube-api-access-895dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:12.140866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.140815 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:12.141412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.141387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:12.141562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.141506 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:12.142193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.142172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy" pod "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" (UID: "70335f13-a25e-4c0b-b6c4-e11b5c20d3bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240866 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240909 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240928 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240942 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240957 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240974 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895dw\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-kube-api-access-895dw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.240992 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.241009 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.249880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.241024 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:12.302505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.302468 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:12.520590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.519542 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66a7573-7cb2-4869-9031-2e64172c16ef" path="/var/lib/kubelet/pods/e66a7573-7cb2-4869-9031-2e64172c16ef/volumes" Nov 01 22:05:12.735429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.735340 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" Nov 01 22:05:12.735429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.735368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc" event={"ID":"3aaa9694-d751-4f13-bd84-6a6d714d3a7e","Type":"ContainerDied","Data":"7e6d9c0400902fbc214f6a1200444d29abaf2f22d152d09c243b7d39c2c491be"} Nov 01 22:05:12.735927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.735429 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e6d9c0400902fbc214f6a1200444d29abaf2f22d152d09c243b7d39c2c491be" Nov 01 22:05:12.737516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.737492 5036 generic.go:356] "Generic (PLEG): container finished" podID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerID="e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70" exitCode=0 Nov 01 22:05:12.737666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.737564 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" Nov 01 22:05:12.737731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.737674 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerDied","Data":"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70"} Nov 01 22:05:12.737731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.737724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn" event={"ID":"70335f13-a25e-4c0b-b6c4-e11b5c20d3bf","Type":"ContainerDied","Data":"1eb20b7216dfdecb145b937088284d0bb388649803267f165ceb3c641188f692"} Nov 01 22:05:12.737837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.737758 5036 scope.go:122] "RemoveContainer" containerID="e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70" Nov 01 22:05:12.739638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.739623 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6956-a75e13" Nov 01 22:05:12.750317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.750298 5036 scope.go:122] "RemoveContainer" containerID="e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582" Nov 01 22:05:12.762457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.762419 5036 scope.go:122] "RemoveContainer" containerID="e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70" Nov 01 22:05:12.762767 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:12.762749 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70\": container with ID starting with e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70 not found: ID does not exist" containerID="e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70" Nov 01 22:05:12.762819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.762775 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70"} err="failed to get container status \"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70\": rpc error: code = NotFound desc = could not find container \"e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70\": container with ID starting with e2a04eede98c184665056150214bce54472b992a7c0515316a0bff00d6754a70 not found: ID does not exist" Nov 01 22:05:12.762819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.762793 5036 scope.go:122] "RemoveContainer" containerID="e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582" Nov 01 22:05:12.763017 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:12.763000 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582\": container with ID starting with e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582 not found: ID does not exist" containerID="e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582" Nov 01 22:05:12.763063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.763021 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582"} err="failed to get container status \"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582\": rpc error: code = NotFound desc = could not find container \"e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582\": container with ID starting with e596075160f21f3417a4f1af3151b55fe73e1e8eb07ddf7ccd55069144440582 not found: ID does not exist" Nov 01 22:05:12.763063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.763034 5036 scope.go:122] "RemoveContainer" containerID="51958886e319ff1ada9f91a306275dcec533b9716979a8eebde7a0e994daee66" Nov 01 22:05:12.765196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.765175 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:05:12.771946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.771926 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b1b5598d-openshift-default-6ddb6cbb5c-xqmkn"] Nov 01 22:05:12.778244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:12.778218 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5"] Nov 01 22:05:12.778816 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:05:12.778797 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ae8d5f_4722_4624_94cc_7d0c4e1f1d9e.slice/crio-d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee WatchSource:0}: Error finding container d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee: Status 404 returned error can't find the container with id d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee Nov 01 22:05:13.133738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:13.133693 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:13.379309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:13.379272 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:13.562471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:13.562064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:13.838206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:13.838098 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" event={"ID":"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e","Type":"ContainerStarted","Data":"dedda617f8a3be6303d53980e7dadfd744ff3e3122794ffb4689031babdf0cdf"} Nov 01 22:05:13.838206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:13.838150 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" event={"ID":"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e","Type":"ContainerStarted","Data":"d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee"} Nov 01 22:05:14.261055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:14.260969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:14.393249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:14.393208 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:14.630668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:14.630617 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" path="/var/lib/kubelet/pods/70335f13-a25e-4c0b-b6c4-e11b5c20d3bf/volumes" Nov 01 22:05:14.845901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:14.845857 5036 generic.go:356] "Generic (PLEG): container finished" podID="f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" containerID="dedda617f8a3be6303d53980e7dadfd744ff3e3122794ffb4689031babdf0cdf" exitCode=0 Nov 01 22:05:14.846354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:14.845905 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" event={"ID":"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e","Type":"ContainerDied","Data":"dedda617f8a3be6303d53980e7dadfd744ff3e3122794ffb4689031babdf0cdf"} Nov 01 22:05:15.340522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:15.339018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:16.122246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.122156 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:16.221470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.209845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:16.252117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.247005 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\" (UniqueName: \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\") pod \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " Nov 01 22:05:16.252117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.247125 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\") pod \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " Nov 01 22:05:16.252117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.247265 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\") pod \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\" (UID: \"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e\") " Nov 01 22:05:16.252117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.249415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca" pod "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" (UID: "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:16.252117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.250927 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca" pod "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" (UID: "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:16.254495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.253403 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr" pod "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" (UID: "f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e"). InnerVolumeSpecName "kube-api-access-4lfdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:16.350175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.350137 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:16.350175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.350176 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:16.350393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.350194 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lfdr\" (UniqueName: \"kubernetes.io/projected/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e-kube-api-access-4lfdr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:16.834862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.834830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:16.932910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.919888 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" event={"ID":"f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e","Type":"ContainerDied","Data":"d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee"} Nov 01 22:05:16.932910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.919957 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cc7e676cd205ce8c7f0d198ce19062580109955f8f568eaf5efc2abe58baee" Nov 01 22:05:16.932910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:16.920049 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5" Nov 01 22:05:17.526710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:17.526672 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:18.350714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:18.350678 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:18.753161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:18.753066 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:18.903063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:18.903028 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:19.059466 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:19.049967 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:05:19.524516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:19.524428 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:20.367312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:20.367227 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:20.887257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:20.887216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:21.043476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.040965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:21.255567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.252097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:21.332604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.332534 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc"] Nov 01 22:05:21.346149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.346115 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21dd16de-p77zc"] Nov 01 22:05:21.462527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.461171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:21.716434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:21.716370 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:22.083010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:22.077634 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:22.201754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:22.201714 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:22.565250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:22.564868 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaa9694-d751-4f13-bd84-6a6d714d3a7e" path="/var/lib/kubelet/pods/3aaa9694-d751-4f13-bd84-6a6d714d3a7e/volumes" Nov 01 22:05:22.844808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:22.844711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:23.643467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:23.642623 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:24.353404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:24.353342 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:25.156129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:25.156089 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:25.835040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:25.835000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:25.885263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:25.885189 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:58972->10.132.0.175:15021: read: connection reset by peer" start-of-body= Nov 01 22:05:25.886793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:25.885427 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:58972->10.132.0.175:15021: read: connection reset by peer" Nov 01 22:05:26.084353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.084256 5036 generic.go:356] "Generic (PLEG): container finished" podID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerID="3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef" exitCode=0 Nov 01 22:05:26.085258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.084349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerDied","Data":"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef"} Nov 01 22:05:26.085258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.084971 5036 scope.go:122] "RemoveContainer" containerID="3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef" Nov 01 22:05:26.087275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.087239 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:26.087416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.087331 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:26.087416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.087374 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:26.737530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.734293 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:05:26.766565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.766359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:26.896310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:26.896155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:27.097629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:27.097584 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerStarted","Data":"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b"} Nov 01 22:05:27.627942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:27.627903 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:27.802432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:27.802396 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:05:27.803175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:27.803147 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:27.803364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:27.803336 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:28.030987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.030952 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947"] Nov 01 22:05:28.031694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031676 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa9694-d751-4f13-bd84-6a6d714d3a7e" containerName="revoke-keys" Nov 01 22:05:28.031694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031695 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa9694-d751-4f13-bd84-6a6d714d3a7e" containerName="revoke-keys" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031710 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" containerName="revoke-keys" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031716 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" containerName="revoke-keys" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031730 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031737 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031762 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.031816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031768 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.032011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031877 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.032011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031890 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70335f13-a25e-4c0b-b6c4-e11b5c20d3bf" containerName="istio-proxy" Nov 01 22:05:28.032011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031900 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa9694-d751-4f13-bd84-6a6d714d3a7e" containerName="revoke-keys" Nov 01 22:05:28.032011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.031906 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" containerName="revoke-keys" Nov 01 22:05:28.035762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.035745 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.113494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.104703 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:28.128089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.127965 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947"] Nov 01 22:05:28.133159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.133125 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:28.141306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.141283 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:28.141418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.141348 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:05:28.166326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.166266 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.167393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.167365 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.167556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.167479 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps84\" (UniqueName: \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.269202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.269165 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.269380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.269233 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps84\" (UniqueName: \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.269510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.269483 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.270142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.270123 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.272124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.272103 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.315659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.311321 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:28.318504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.317896 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps84\" (UniqueName: \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\") pod \"maas-api-revoke-sub-a68a8505-wz947\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.350774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.350748 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:28.523644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.523590 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:05:28.618605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.615907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:28.810468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.796739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:28.829467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.826094 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:28.829467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.826156 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:28.918465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:28.906285 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947"] Nov 01 22:05:29.153727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:29.153545 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" event={"ID":"4ead712b-a22d-4252-a516-bd6792e4de42","Type":"ContainerStarted","Data":"8619983c4e75e0e1ecacea125168da746358b98db1e2f4cb25f4e17be7d84b6c"} Nov 01 22:05:29.819331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:29.815593 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:29.819331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:29.815656 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:29.852927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:29.852834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:30.147629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.147534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:30.165125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.165084 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" event={"ID":"4ead712b-a22d-4252-a516-bd6792e4de42","Type":"ContainerStarted","Data":"54a39a7b4a0e3e10b858c35a80ba2212ec99eb43dff9eebeb874f997ce768856"} Nov 01 22:05:30.209471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.195640 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" podStartSLOduration=3.195621336 podStartE2EDuration="3.195621336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:30.194683733 +0000 UTC m=+1283.887664222" watchObservedRunningTime="2026-11-01 22:05:30.195621336 +0000 UTC m=+1283.888601823" Nov 01 22:05:30.597487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.597425 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:30.807467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.806750 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:30.807467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.806816 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:30.849527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:30.848821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:31.015304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.007359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:31.382468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.381457 5036 generic.go:356] "Generic (PLEG): container finished" podID="4ead712b-a22d-4252-a516-bd6792e4de42" containerID="54a39a7b4a0e3e10b858c35a80ba2212ec99eb43dff9eebeb874f997ce768856" exitCode=0 Nov 01 22:05:31.382468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.381760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" event={"ID":"4ead712b-a22d-4252-a516-bd6792e4de42","Type":"ContainerDied","Data":"54a39a7b4a0e3e10b858c35a80ba2212ec99eb43dff9eebeb874f997ce768856"} Nov 01 22:05:31.816480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.802875 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:31.816480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.807710 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:31.816480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:31.807783 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:32.302599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.302525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:32.377266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.377232 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:32.684843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.684705 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:32.707951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.707897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:32.814468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.806164 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:32.814468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.806239 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:32.847738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.847674 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:32.881711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.881657 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:32.990524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:32.983544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:33.375597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.372837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": context deadline exceeded" Nov 01 22:05:33.375597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.373183 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:33.375597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.373412 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:33.385876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.385842 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:05:33.404730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.387336 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:05:33.563103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.563057 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:33.758999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.758906 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:05:33.774968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.774942 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:33.851608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.851554 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:05:33.889086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.884758 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:33.915571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.897268 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:05:33.935702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.915814 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:33.936282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.936254 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:33.951154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.951117 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.025109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:33.971762 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:34.066973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.025698 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:05:34.374817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.374706 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.402643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.402577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.403930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.082014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:34.404275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.111204 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.405169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.145570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.405631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.145647 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.405777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.204418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca" pod "4ead712b-a22d-4252-a516-bd6792e4de42" (UID: "4ead712b-a22d-4252-a516-bd6792e4de42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:34.405893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.146656 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\") pod \"4ead712b-a22d-4252-a516-bd6792e4de42\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " Nov 01 22:05:34.449628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.449592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\") pod \"4ead712b-a22d-4252-a516-bd6792e4de42\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " Nov 01 22:05:34.450076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.450055 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\" (UniqueName: \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\") pod \"4ead712b-a22d-4252-a516-bd6792e4de42\" (UID: \"4ead712b-a22d-4252-a516-bd6792e4de42\") " Nov 01 22:05:34.451545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.451519 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:34.480937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.028623 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:34.485629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.485583 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:34.532486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.505673 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca" pod "4ead712b-a22d-4252-a516-bd6792e4de42" (UID: "4ead712b-a22d-4252-a516-bd6792e4de42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:34.571650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.527136 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84" pod "4ead712b-a22d-4252-a516-bd6792e4de42" (UID: "4ead712b-a22d-4252-a516-bd6792e4de42"). InnerVolumeSpecName "kube-api-access-hps84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:34.571905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.570649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:05:34.618598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.618540 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hps84\" (UniqueName: \"kubernetes.io/projected/4ead712b-a22d-4252-a516-bd6792e4de42-kube-api-access-hps84\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:34.618835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.618816 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead712b-a22d-4252-a516-bd6792e4de42-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:34.785471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.678494 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:34.794675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.787331 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:05:34.794675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.787401 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:34.794675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.787420 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 22:05:34.815006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.814975 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 22:05:34.815271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.815253 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 22:05:34.815639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.815607 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" containerID="cri-o://a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb" gracePeriod=30 Nov 01 22:05:34.816260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.816209 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" containerID="cri-o://a8efc8422551f279832b6e3d47060d5be95809267aa54cba405ab7deb282b2bc" gracePeriod=30 Nov 01 22:05:34.817199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.817087 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" containerID="cri-o://7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231" gracePeriod=30 Nov 01 22:05:34.824347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.823082 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:34.824347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.823294 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:34.848716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.847934 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.066921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.910930 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.066921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.927397 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:05:35.066921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.928547 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4ead712b-a22d-4252-a516-bd6792e4de42" containerName="revoke-keys" Nov 01 22:05:35.066921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.928569 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ead712b-a22d-4252-a516-bd6792e4de42" containerName="revoke-keys" Nov 01 22:05:35.066921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:34.928790 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ead712b-a22d-4252-a516-bd6792e4de42" containerName="revoke-keys" Nov 01 22:05:35.155468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.109179 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:05:35.224600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.145840 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:35.224600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.149243 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:05:35.224600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.218330 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:05:35.224600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.218591 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.231632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.231600 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edd32a\"/\"test-model-edd32a-kserve-self-signed-certs\"" Nov 01 22:05:35.237585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.232071 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edd32a\"/\"kube-root-ca.crt\"" Nov 01 22:05:35.238119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.233283 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edd32a\"/\"openshift-service-ca.crt\"" Nov 01 22:05:35.238273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.233388 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edd32a\"/\"default-dockercfg-vbtxw\"" Nov 01 22:05:35.350567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.339546 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.350567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.339794 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:05:35.350567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.340995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:35.370886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.370848 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.372654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.372627 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:05:35.375460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.375418 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.404257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.404223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw64\" (UniqueName: \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.404872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.404846 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.405248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.405228 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.405396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.405380 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.405624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.405606 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.405785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.405765 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.407931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.407903 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:35.423353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.410502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:05:35.427809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.410532 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": context deadline exceeded" Nov 01 22:05:35.433283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.413772 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" Nov 01 22:05:35.452132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.452106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947" event={"ID":"4ead712b-a22d-4252-a516-bd6792e4de42","Type":"ContainerDied","Data":"8619983c4e75e0e1ecacea125168da746358b98db1e2f4cb25f4e17be7d84b6c"} Nov 01 22:05:35.452335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.452315 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8619983c4e75e0e1ecacea125168da746358b98db1e2f4cb25f4e17be7d84b6c" Nov 01 22:05:35.452665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.452646 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:05:35.452872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.452859 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.517600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw64\" (UniqueName: \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.517656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.517886 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.517919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.521164 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.522915 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.539470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.523675 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.571586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.564589 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.571586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.564709 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.594808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.594732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw64\" (UniqueName: \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.617577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.615697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.813486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.792282 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\") pod \"test-model-edd32a-kserve-75ff95445-jbq4p\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:35.830465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.816409 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:35.831541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.831503 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:35.861620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.861575 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:35.862684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.862657 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:05:35.889038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.888946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:35.991594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:35.991559 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:05:36.015425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.012810 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:36.200577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.140214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:36.453327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.183704 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.463705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.183744 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.475272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.200864 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.486219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.351607 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.785486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.372420 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.785486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.379690 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:36.785486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.428432 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.785486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.428495 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.868563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.868493 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:05:36.890907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.890655 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:36.966678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:36.941943 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:05:37.002773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.002725 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:37.012759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.012687 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:37.024478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.013052 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:37.112760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.036338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:37.116501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.114597 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:37.116501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.114696 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:05:37.251141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.250960 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:37.309629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.309566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:37.477349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.475882 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:05:37.571805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.568124 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:37.802328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.802291 5036 patch_prober.go:29] interesting pod/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:05:37.802619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.802586 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 22:05:37.821853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.821791 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:05:37.833643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.833615 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:37.847244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.846969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" event={"ID":"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1","Type":"ContainerStarted","Data":"c5cccc317d9dc724666355524caee76097906d9fc279455d2810fe5c8671c96f"} Nov 01 22:05:37.960476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:37.930670 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:05:38.023647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.023590 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:05:38.024076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.024042 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:05:38.027569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.024381 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:38.027569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.024706 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:05:38.027569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.024998 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:38.027569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.025298 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:38.167471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.048935 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:05:38.168046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.168017 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:05:38.194526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.074272 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:38.389809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.378931 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:05:38.500521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.483919 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:38.610733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.588338 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:05:38.612131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.612093 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:05:38.647667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.647554 5036 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod506c1455-71ab-42d7-9740-f45b771dd6a7"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod506c1455-71ab-42d7-9740-f45b771dd6a7] : Timed out while waiting for systemd to remove kubepods-burstable-pod506c1455_71ab_42d7_9740_f45b771dd6a7.slice" Nov 01 22:05:38.648095 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:38.648054 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod506c1455-71ab-42d7-9740-f45b771dd6a7] : unable to destroy cgroup paths for cgroup [kubepods burstable pod506c1455-71ab-42d7-9740-f45b771dd6a7] : Timed out while waiting for systemd to remove kubepods-burstable-pod506c1455_71ab_42d7_9740_f45b771dd6a7.slice" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" podUID="506c1455-71ab-42d7-9740-f45b771dd6a7" Nov 01 22:05:38.689161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:38.689065 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:39.001141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.001043 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:39.017637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.004189 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:39.046959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041607 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:05:39.047081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041650 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:39.047081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041673 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:39.047081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041694 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:39.047081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041713 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:39.047325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.041733 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:39.160467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.159838 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:05:39.173886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.162169 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:05:39.221046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.209142 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:05:39.288682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.288649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:05:39.314468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.314306 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp" Nov 01 22:05:39.391053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.390523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:05:39.458569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.458518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": context deadline exceeded" Nov 01 22:05:39.843470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.838709 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:39.943466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:39.936688 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:40.018468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.001578 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:05:40.018468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.002089 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:05:40.340267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.340229 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.348643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.463808 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.362991 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.464658 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.464727 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.363079 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.363020 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.363040 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:40.479554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.363060 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:05:40.502471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.363097 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:40.502471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.412768 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:05:40.502471 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:40.495622 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb78bdd_d5e7_4503_9e44_9e69cbb447f1.slice/crio-bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6ca82_79fd_4d85_a62e_d6e3310e7852.slice/crio-7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:05:40.502471 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:40.499327 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-conmon-a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:05:40.516545 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:40.503909 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-conmon-a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6ca82_79fd_4d85_a62e_d6e3310e7852.slice/crio-7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6ca82_79fd_4d85_a62e_d6e3310e7852.slice/crio-conmon-7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:05:40.520487 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:40.518124 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6ca82_79fd_4d85_a62e_d6e3310e7852.slice/crio-conmon-7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d289b4_a752_4cbf_b4a2_70bd1de0d5bc.slice/crio-conmon-a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:05:40.529502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.444477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:40.851248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.851187 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 01 22:05:40.851690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.851306 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:40.931525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:40.926145 5036 generic.go:356] "Generic (PLEG): container finished" podID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerID="7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231" exitCode=0 Nov 01 22:05:41.173469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.167570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" event={"ID":"23c6ca82-79fd-4d85-a62e-d6e3310e7852","Type":"ContainerDied","Data":"7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231"} Nov 01 22:05:41.173469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.167754 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:05:41.173469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.167826 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:05:41.173469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.172151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:41.173469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.172668 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:41.238856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.238819 5036 generic.go:356] "Generic (PLEG): container finished" podID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerID="a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb" exitCode=0 Nov 01 22:05:41.240539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.240493 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" event={"ID":"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc","Type":"ContainerDied","Data":"a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb"} Nov 01 22:05:41.243270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.242301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.467836 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.468014 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.468154 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.468296 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.468493 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:41.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.468959 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:41.489694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.489653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:05:41.577471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.542863 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:41.614499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.614467 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:05:41.692468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:41.690514 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:42.009998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.009970 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:05:42.021932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.021898 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:42.170751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.110921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:42.176677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.176641 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:42.192480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.190586 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:05:42.466016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.465713 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" containerID="cri-o://6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8" gracePeriod=45 Nov 01 22:05:42.470697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.470121 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:42.492183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.491289 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.556887 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.556948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557030 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557159 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557221 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557272 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557317 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557368 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.558462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.557500 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\") pod \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\" (UID: \"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc\") " Nov 01 22:05:42.559049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.558997 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:05:42.559049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.559025 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" event={"ID":"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1","Type":"ContainerStarted","Data":"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8"} Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.559079 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.559118 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.559155 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.559206 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.561165 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.562503 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.562759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.565642 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.566992 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.572398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.567052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:42.586329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.586295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" event={"ID":"c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc","Type":"ContainerDied","Data":"cd3a95ac2ac95f8c7bd097ccb63a15256c9d25a1547f17f6728bdfac7e197af9"} Nov 01 22:05:42.586499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.586362 5036 scope.go:122] "RemoveContainer" containerID="a124470dfc0150baa9ac42058cf2bd503b70631b99acca09eb7b7ed85e3151eb" Nov 01 22:05:42.586564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.586537 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6" Nov 01 22:05:42.588430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.588379 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podStartSLOduration=9.588363557 podStartE2EDuration="9.588363557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:42.566235598 +0000 UTC m=+1296.259216081" watchObservedRunningTime="2026-11-01 22:05:42.588363557 +0000 UTC m=+1296.281344047" Nov 01 22:05:42.595189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.592746 5036 generic.go:356] "Generic (PLEG): container finished" podID="0ed97d34-a363-477c-8113-57875f045680" containerID="a8efc8422551f279832b6e3d47060d5be95809267aa54cba405ab7deb282b2bc" exitCode=0 Nov 01 22:05:42.595189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.594763 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" event={"ID":"0ed97d34-a363-477c-8113-57875f045680","Type":"ContainerDied","Data":"a8efc8422551f279832b6e3d47060d5be95809267aa54cba405ab7deb282b2bc"} Nov 01 22:05:42.595288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.595230 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "kube-api-access-zzr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.595513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.595489 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:42.595637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.595574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.603897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.603754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:42.606832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.606793 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token" pod "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" (UID: "c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.646564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.646535 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 22:05:42.675312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675285 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675310 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675321 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675330 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675339 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzr2t\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-kube-api-access-zzr2t\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675350 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675359 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675368 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.675538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.675377 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.677242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.677168 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 22:05:42.776367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776402 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.776570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776467 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.776691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776669 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776706 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.776802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776757 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776836 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776879 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.776978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.777078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.776995 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.777128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.777128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\") pod \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\" (UID: \"23c6ca82-79fd-4d85-a62e-d6e3310e7852\") " Nov 01 22:05:42.777235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777213 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.777291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.777346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777333 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.777403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777364 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.777476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.777418 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\") pod \"0ed97d34-a363-477c-8113-57875f045680\" (UID: \"0ed97d34-a363-477c-8113-57875f045680\") " Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.779675 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.779879 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.780065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.780250 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.780640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.781006 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:42.789144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.781511 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.803151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.795227 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.810166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.802069 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.810166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.807575 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.810166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.807837 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:42.810631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.810565 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.811521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.811215 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "kube-api-access-nlchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.811521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.811252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.811521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.811362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.811521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.811479 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:42.815068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.815030 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy" pod "0ed97d34-a363-477c-8113-57875f045680" (UID: "0ed97d34-a363-477c-8113-57875f045680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:42.818671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.818645 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r" pod "23c6ca82-79fd-4d85-a62e-d6e3310e7852" (UID: "23c6ca82-79fd-4d85-a62e-d6e3310e7852"). InnerVolumeSpecName "kube-api-access-5b92r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884857 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884894 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlchb\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-kube-api-access-nlchb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884915 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884932 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884948 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884964 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b92r\" (UniqueName: \"kubernetes.io/projected/23c6ca82-79fd-4d85-a62e-d6e3310e7852-kube-api-access-5b92r\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884979 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.884993 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885007 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885020 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885035 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885048 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885057 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885067 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed97d34-a363-477c-8113-57875f045680-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885076 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed97d34-a363-477c-8113-57875f045680-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885085 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23c6ca82-79fd-4d85-a62e-d6e3310e7852-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885098 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed97d34-a363-477c-8113-57875f045680-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.885160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.885112 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed97d34-a363-477c-8113-57875f045680-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.991253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:42.991218 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 22:05:43.010149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.010040 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-b2bc6"] Nov 01 22:05:43.022363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.022297 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:43.044376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.044038 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.089396 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.089574 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.089764 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.089795 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.089843 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.090166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.090032 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\" (UniqueName: \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\") pod \"4f756862-7959-4a07-b087-4ac08be816b3\" (UID: \"4f756862-7959-4a07-b087-4ac08be816b3\") " Nov 01 22:05:43.094066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.091388 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:43.094066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.091622 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:43.094066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.091936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:43.101785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.101739 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:43.101785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.101769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "kube-api-access-rps75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:43.107721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.107676 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs" pod "4f756862-7959-4a07-b087-4ac08be816b3" (UID: "4f756862-7959-4a07-b087-4ac08be816b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:43.192342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192265 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rps75\" (UniqueName: \"kubernetes.io/projected/4f756862-7959-4a07-b087-4ac08be816b3-kube-api-access-rps75\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.192342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192312 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.192342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192325 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.192342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192339 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.192342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192350 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f756862-7959-4a07-b087-4ac08be816b3-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.192679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.192364 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f756862-7959-4a07-b087-4ac08be816b3-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:43.433207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.433115 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:43.628408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.628369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" event={"ID":"23c6ca82-79fd-4d85-a62e-d6e3310e7852","Type":"ContainerDied","Data":"0e0b86e7dacb8f6a80b6c6c2007e16ea46d445f9749b172b06a49ef0865c6151"} Nov 01 22:05:43.628753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.628733 5036 scope.go:122] "RemoveContainer" containerID="7e05a6aae07f6777234640220f45d2153ddf1d600bf1bdd593e9e01339fcf231" Nov 01 22:05:43.629055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.629037 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs" Nov 01 22:05:43.635482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.635169 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" Nov 01 22:05:43.635651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.635625 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4" event={"ID":"0ed97d34-a363-477c-8113-57875f045680","Type":"ContainerDied","Data":"ca52e0dacdea263674d4eebfab12d2a4f7eb9021ce127ba0f95bdae445068bef"} Nov 01 22:05:43.641261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.641233 5036 generic.go:356] "Generic (PLEG): container finished" podID="4f756862-7959-4a07-b087-4ac08be816b3" containerID="6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8" exitCode=0 Nov 01 22:05:43.641475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.641433 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" event={"ID":"4f756862-7959-4a07-b087-4ac08be816b3","Type":"ContainerDied","Data":"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8"} Nov 01 22:05:43.642460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.642419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" event={"ID":"4f756862-7959-4a07-b087-4ac08be816b3","Type":"ContainerDied","Data":"84ba94477b8fb2f4a775da62cba0a1f77d0baa5ffbe49d8e4d0f2666b2f38a2e"} Nov 01 22:05:43.642601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.642183 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg" Nov 01 22:05:43.649203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.649183 5036 scope.go:122] "RemoveContainer" containerID="a8efc8422551f279832b6e3d47060d5be95809267aa54cba405ab7deb282b2bc" Nov 01 22:05:43.662958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.662937 5036 scope.go:122] "RemoveContainer" containerID="6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8" Nov 01 22:05:43.676042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.676011 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 22:05:43.681500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.681436 5036 scope.go:122] "RemoveContainer" containerID="6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8" Nov 01 22:05:43.681834 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:43.681810 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8\": container with ID starting with 6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8 not found: ID does not exist" containerID="6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8" Nov 01 22:05:43.681910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.681847 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8"} err="failed to get container status \"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8\": rpc error: code = NotFound desc = could not find container \"6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8\": container with ID starting with 6af6e1d2e042ae88d1670336b25076e1f84769195b798e83b461f806c11fa9f8 not found: ID does not exist" Nov 01 22:05:43.682950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.682926 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fmpxs"] Nov 01 22:05:43.696904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.696849 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:05:43.704044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.704023 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-21dd16de/e2e-lifecycle-model-21dd16de-kserve-7c5458b749-c5dfg"] Nov 01 22:05:43.712580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.712540 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 22:05:43.719725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.719701 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-6mzs4"] Nov 01 22:05:43.897683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.897643 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:43.897949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:43.897915 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:44.263786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:44.263650 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:44.446526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:44.446490 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed97d34-a363-477c-8113-57875f045680" path="/var/lib/kubelet/pods/0ed97d34-a363-477c-8113-57875f045680/volumes" Nov 01 22:05:44.447051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:44.447032 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" path="/var/lib/kubelet/pods/23c6ca82-79fd-4d85-a62e-d6e3310e7852/volumes" Nov 01 22:05:44.447623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:44.447602 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f756862-7959-4a07-b087-4ac08be816b3" path="/var/lib/kubelet/pods/4f756862-7959-4a07-b087-4ac08be816b3/volumes" Nov 01 22:05:44.448621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:44.448602 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" path="/var/lib/kubelet/pods/c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc/volumes" Nov 01 22:05:45.029740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:45.029702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:45.617701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:45.617646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:45.811466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:45.800249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:46.027523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.023420 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:46.143467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.125904 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:46.193468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.184001 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:46.338527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.332136 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp"] Nov 01 22:05:46.342988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.341075 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d506c6-z98tp"] Nov 01 22:05:46.367459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.367404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:46.447951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.447906 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506c1455-71ab-42d7-9740-f45b771dd6a7" path="/var/lib/kubelet/pods/506c1455-71ab-42d7-9740-f45b771dd6a7/volumes" Nov 01 22:05:46.541466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.510836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:46.707650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.707553 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:05:46.851919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.851876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:46.868589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.868542 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:46.868747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:46.868617 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:47.197295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:47.197253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:47.368631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:47.368596 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:47.938041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:47.937954 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:48.065233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.065193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:48.336668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.336583 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:05:48.337666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337643 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" Nov 01 22:05:48.337754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337668 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" Nov 01 22:05:48.337754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337736 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" Nov 01 22:05:48.337754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337744 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" Nov 01 22:05:48.337915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337769 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" Nov 01 22:05:48.337915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337777 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" Nov 01 22:05:48.337915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337796 5036 cpu_manager.go:414] "containerMap: removing container" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" Nov 01 22:05:48.337915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.337803 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" Nov 01 22:05:48.338115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.338011 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f756862-7959-4a07-b087-4ac08be816b3" containerName="main" Nov 01 22:05:48.338115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.338028 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c6ca82-79fd-4d85-a62e-d6e3310e7852" containerName="istio-proxy" Nov 01 22:05:48.338115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.338046 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed97d34-a363-477c-8113-57875f045680" containerName="istio-proxy" Nov 01 22:05:48.338115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.338064 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d289b4-a752-4cbf-b4a2-70bd1de0d5bc" containerName="istio-proxy" Nov 01 22:05:48.342538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.342515 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.355568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.354197 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18657f-openshift-default-dockercfg-6hqh2\"" Nov 01 22:05:48.367541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.367511 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:05:48.433522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.419828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525520 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525572 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpnz\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525672 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.529467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.525782 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.693042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.674001 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.693305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.693271 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.693550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.693519 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.698549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.698521 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpnz\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.698754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.698740 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.698895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.698883 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.699015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.699003 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.704782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.704754 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.705042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.705025 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.720839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.720808 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.722697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.711643 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.753689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.712183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.764183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.722036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.764412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.710626 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.896465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.874493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpnz\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.897417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.897353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.928400 5036 generic.go:356] "Generic (PLEG): container finished" podID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerID="650ae3df6ea3cfeca4dad308e68a67d5fe43bfecd75cf543c9b8c64695813132" exitCode=0 Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.928760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerDied","Data":"650ae3df6ea3cfeca4dad308e68a67d5fe43bfecd75cf543c9b8c64695813132"} Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.929276 5036 scope.go:122] "RemoveContainer" containerID="650ae3df6ea3cfeca4dad308e68a67d5fe43bfecd75cf543c9b8c64695813132" Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.933412 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.933639 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.934840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.933706 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.938502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.937368 5036 generic.go:356] "Generic (PLEG): container finished" podID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerID="d661453ba9a870dc2e9dd93b3293b9e60a244afd9eb8b3b272b74979f8bcb578" exitCode=0 Nov 01 22:05:48.938502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.937431 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerDied","Data":"d661453ba9a870dc2e9dd93b3293b9e60a244afd9eb8b3b272b74979f8bcb578"} Nov 01 22:05:48.938502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.937916 5036 scope.go:122] "RemoveContainer" containerID="d661453ba9a870dc2e9dd93b3293b9e60a244afd9eb8b3b272b74979f8bcb578" Nov 01 22:05:48.944820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.943041 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.946492 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:05:48.946315 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:05:48.951071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.951033 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\") pod \"e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:48.952624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.951955 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.952624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.952022 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.952624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.952053 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:48.998489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:48.994622 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:49.045471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.041270 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:49.057947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.052697 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 22:05:49.057947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.053002 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" containerID="cri-o://28785ef7ebdc8dfc27bb9f90e569fb644e0b4327743dca70e9885db2fddcaad6" gracePeriod=30 Nov 01 22:05:49.201482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.200098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:49.551865 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:05:49.551659 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da3113_b386_46e0_9ad1_cd0f3915d26d.slice/crio-840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf WatchSource:0}: Error finding container 840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf: Status 404 returned error can't find the container with id 840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf Nov 01 22:05:49.552085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.552055 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:05:49.555587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.555236 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:49.555587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.555294 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:49.555587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.555322 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:05:49.828530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.828391 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:49.993095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:49.993057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:50.011579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.011522 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerStarted","Data":"06a46c50e7c6bbc5a12f32312687ea50fe4c1e35373bc22debd55268f618fbc2"} Nov 01 22:05:50.021101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.021064 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerStarted","Data":"41515f3a543973c4695ee1967d9eeb6fb2ba56b5d6554817f3f571d508cb939f"} Nov 01 22:05:50.033940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.033900 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerStarted","Data":"b2221105533cf001c0e12b39ea645ba2eea7939d0bae75acadb001377e057166"} Nov 01 22:05:50.036205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.035242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerStarted","Data":"840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf"} Nov 01 22:05:50.167327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.167194 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podStartSLOduration=2.167172166 podStartE2EDuration="2.167172166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:50.163022666 +0000 UTC m=+1303.856003151" watchObservedRunningTime="2026-11-01 22:05:50.167172166 +0000 UTC m=+1303.860152654" Nov 01 22:05:50.400470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.395066 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:05:50.466279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.466183 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:50.466564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.466532 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:50.863467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.861529 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:05:50.863467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.861686 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:50.863467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:50.861746 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:51.007078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.007048 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:51.027515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.026729 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:51.074558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.074527 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:51.089495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.089463 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:05:51.158565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.156005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:51.404661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.404628 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:51.417679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.404953 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:51.473467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.472692 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:05:51.852879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.852842 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:51.853155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:51.853120 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:52.030468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.028476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:52.401468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.400016 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:52.401468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.400071 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:52.424479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.402199 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:05:52.424479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.402262 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:05:52.485467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.471017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:52.635471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.624167 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:52.855469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.855174 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:52.855469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.855251 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:52.912018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:52.910903 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:53.396467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.395799 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:53.396467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.395863 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:53.662548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.661514 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:05:53.668209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.667811 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:53.680464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.673077 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:05:53.680464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.673195 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.721818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.720736 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3577a0\"/\"test-model-3577a0-kserve-self-signed-certs\"" Nov 01 22:05:53.721818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.720890 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3577a0\"/\"kube-root-ca.crt\"" Nov 01 22:05:53.721818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.721039 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3577a0\"/\"openshift-service-ca.crt\"" Nov 01 22:05:53.747076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.747051 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3577a0\"/\"default-dockercfg-xcrfv\"" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772217 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772268 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772481 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.782859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.772620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8fs\" (UniqueName: \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:53.942519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.942380 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:54.017488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:53.942437 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:54.115767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.036918 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8fs\" (UniqueName: \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.116365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.116314 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.116617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.116597 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.134789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.134753 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.135101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.135079 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.135241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.135226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.136062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.136034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.148471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.136347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:54.170040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.136491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.225155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.136965 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.225155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.186714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.225155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.204082 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.242069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.226298 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8fs\" (UniqueName: \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\") pod \"test-model-3577a0-kserve-789f498ff7-m7g2k\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.396899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.396472 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:54.396899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.396531 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:54.419494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.413954 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:54.619172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.617593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:54.670413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.670345 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:54.754481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.751347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:05:54.791666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.788274 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:54.791666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.790069 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:54.806853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.806431 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:54.921407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.921057 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:54.921407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.921117 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:54.943438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:54.943030 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:55.077883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.076838 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:55.180981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.180409 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:05:55.321664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.318592 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:55.421387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.395566 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:55.459375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.450910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" event={"ID":"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4","Type":"ContainerStarted","Data":"d647e3bfc970d649c83347d9b0a05dd4186467d50c9a86d01328900fd5b43a83"} Nov 01 22:05:55.459375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.451120 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:55.459375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.451169 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:55.565663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.557196 5036 generic.go:356] "Generic (PLEG): container finished" podID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerID="28785ef7ebdc8dfc27bb9f90e569fb644e0b4327743dca70e9885db2fddcaad6" exitCode=0 Nov 01 22:05:55.565663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.557351 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" event={"ID":"11b12813-1b64-4e6a-844f-ca3a13af79ec","Type":"ContainerDied","Data":"28785ef7ebdc8dfc27bb9f90e569fb644e0b4327743dca70e9885db2fddcaad6"} Nov 01 22:05:55.637513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.629475 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:55.753424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.687608 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:55.758796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.758660 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:55.759150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.758985 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:05:55.759853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.759679 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" containerID="cri-o://af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b" gracePeriod=30 Nov 01 22:05:55.991468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.983870 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:55.991468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.984039 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:55.991468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:55.984083 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:56.068331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.068279 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 22:05:56.227048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.186226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:56.345738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.345640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.346089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.346067 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.346236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.346221 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.346479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.346458 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.346667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.346651 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.346891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.346873 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.347075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.347055 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.347343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.347322 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.352126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.352094 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\") pod \"11b12813-1b64-4e6a-844f-ca3a13af79ec\" (UID: \"11b12813-1b64-4e6a-844f-ca3a13af79ec\") " Nov 01 22:05:56.353784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.353760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:56.353922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.353831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:56.354112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.354083 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:56.407681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.398074 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "kube-api-access-sk5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:56.407681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.398179 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:56.407681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.398701 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:05:56.407681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.402176 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:56.407681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.402243 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:56.440699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.427418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:56.496319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.496273 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:05:56.496846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.496814 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:05:56.499178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.498536 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.499178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.498576 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.499178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.498592 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.499178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.498611 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5qq\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-kube-api-access-sk5qq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.499178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.498626 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.562887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.562849 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:56.642530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.634472 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b12813-1b64-4e6a-844f-ca3a13af79ec-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.645233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.645176 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:56.651813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.651772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:56.659794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.659767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo" pod "11b12813-1b64-4e6a-844f-ca3a13af79ec" (UID: "11b12813-1b64-4e6a-844f-ca3a13af79ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:56.662493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.662427 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" Nov 01 22:05:56.662663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.662641 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz"] Nov 01 22:05:56.663745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.663715 5036 cpu_manager.go:414] "containerMap: removing container" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" Nov 01 22:05:56.663745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.663740 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" Nov 01 22:05:56.664015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.664000 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" containerName="istio-proxy" Nov 01 22:05:56.668909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.668875 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf" event={"ID":"11b12813-1b64-4e6a-844f-ca3a13af79ec","Type":"ContainerDied","Data":"c2ecae6b012607c8e1de390357e55ea274568547c57fbaef48f158b812485ebd"} Nov 01 22:05:56.669083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.668965 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.669301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.669094 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:05:56.669492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.669438 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz"] Nov 01 22:05:56.669689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.669147 5036 scope.go:122] "RemoveContainer" containerID="28785ef7ebdc8dfc27bb9f90e569fb644e0b4327743dca70e9885db2fddcaad6" Nov 01 22:05:56.672639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.672623 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:56.673664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.673646 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:05:56.673769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.673741 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:56.673835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.673793 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:56.760686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.760658 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.760887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.760868 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b12813-1b64-4e6a-844f-ca3a13af79ec-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.761005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.760992 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b12813-1b64-4e6a-844f-ca3a13af79ec-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:56.811705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.811664 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 22:05:56.854852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.853296 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:05:56.854852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.853357 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 22:05:56.870614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.864537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7xg\" (UniqueName: \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.870614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.864631 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.870614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.864730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.878767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.876013 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-psclf"] Nov 01 22:05:56.952177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.952041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:56.974139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.968858 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7xg\" (UniqueName: \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.974139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.971662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.974139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.971835 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:56.990036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.990002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:57.060863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:56.994724 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:57.061155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.002468 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7xg\" (UniqueName: \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\") pod \"maas-api-revoke-keys-e2e-api-b-004180-mwfhz\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:57.172898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.172855 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:05:57.394877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.394840 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:05:57.454496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.451212 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 22:05:57.525892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.525742 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:57.541715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.541691 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz"] Nov 01 22:05:57.545270 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:05:57.545242 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5049bef6_c09d_402e_a301_7c665a20451e.slice/crio-461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2 WatchSource:0}: Error finding container 461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2: Status 404 returned error can't find the container with id 461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2 Nov 01 22:05:57.674364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.674322 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" event={"ID":"5049bef6-c09d-402e-a301-7c665a20451e","Type":"ContainerStarted","Data":"461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2"} Nov 01 22:05:57.676244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.676217 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" event={"ID":"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4","Type":"ContainerStarted","Data":"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13"} Nov 01 22:05:57.702705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.702639 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podStartSLOduration=4.702596979 podStartE2EDuration="4.702596979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:57.695748308 +0000 UTC m=+1311.388728804" watchObservedRunningTime="2026-11-01 22:05:57.702596979 +0000 UTC m=+1311.395577465" Nov 01 22:05:57.754721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.754679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:57.855585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.855488 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:05:57.866706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.863239 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:05:57.866706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.864879 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:05:57.867610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:57.867582 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:05:58.400750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.400718 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:05:58.414897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.414872 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:05:58.441285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.441249 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b12813-1b64-4e6a-844f-ca3a13af79ec" path="/var/lib/kubelet/pods/11b12813-1b64-4e6a-844f-ca3a13af79ec/volumes" Nov 01 22:05:58.596693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.596631 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:58.696555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.692112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" event={"ID":"5049bef6-c09d-402e-a301-7c665a20451e","Type":"ContainerStarted","Data":"3c3ea4abfac61108393bd90bb287aaab1954f59975d4da668c6339af9bd305cd"} Nov 01 22:05:58.696783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.696729 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:05:58.714823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:58.714769 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" podStartSLOduration=2.714754086 podStartE2EDuration="2.714754086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:58.711031371 +0000 UTC m=+1312.404011857" watchObservedRunningTime="2026-11-01 22:05:58.714754086 +0000 UTC m=+1312.407734573" Nov 01 22:05:59.373776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:59.373740 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:05:59.705975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:59.705879 5036 generic.go:356] "Generic (PLEG): container finished" podID="5049bef6-c09d-402e-a301-7c665a20451e" containerID="3c3ea4abfac61108393bd90bb287aaab1954f59975d4da668c6339af9bd305cd" exitCode=0 Nov 01 22:05:59.705975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:05:59.705960 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" event={"ID":"5049bef6-c09d-402e-a301-7c665a20451e","Type":"ContainerDied","Data":"3c3ea4abfac61108393bd90bb287aaab1954f59975d4da668c6339af9bd305cd"} Nov 01 22:06:00.035825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.035787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:00.222411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.221403 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:00.653875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.653835 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5"] Nov 01 22:06:00.658740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.658715 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a19582-tmhp5"] Nov 01 22:06:00.697433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.697399 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:00.844089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.844003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:00.951908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:00.951868 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:06:01.107284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.107252 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\") pod \"5049bef6-c09d-402e-a301-7c665a20451e\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " Nov 01 22:06:01.107527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.107508 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\") pod \"5049bef6-c09d-402e-a301-7c665a20451e\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " Nov 01 22:06:01.107621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.107606 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\" (UniqueName: \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\") pod \"5049bef6-c09d-402e-a301-7c665a20451e\" (UID: \"5049bef6-c09d-402e-a301-7c665a20451e\") " Nov 01 22:06:01.107960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.107929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca" pod "5049bef6-c09d-402e-a301-7c665a20451e" (UID: "5049bef6-c09d-402e-a301-7c665a20451e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:01.108291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.108270 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.109524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.109502 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca" pod "5049bef6-c09d-402e-a301-7c665a20451e" (UID: "5049bef6-c09d-402e-a301-7c665a20451e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:01.110078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.110054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg" pod "5049bef6-c09d-402e-a301-7c665a20451e" (UID: "5049bef6-c09d-402e-a301-7c665a20451e"). InnerVolumeSpecName "kube-api-access-2c7xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:01.174003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.173937 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:01.210186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.210159 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5049bef6-c09d-402e-a301-7c665a20451e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.210186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.210183 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7xg\" (UniqueName: \"kubernetes.io/projected/5049bef6-c09d-402e-a301-7c665a20451e-kube-api-access-2c7xg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.481192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.481161 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:06:01.616048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616018 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616103 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616130 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616151 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616347 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616477 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:01.616544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616496 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.616651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:01.616651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616681 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\") pod \"ed15519a-d1c2-4733-9182-9cdedf40f833\" (UID: \"ed15519a-d1c2-4733-9182-9cdedf40f833\") " Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.616901 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617038 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617116 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617517 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617541 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617564 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617579 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15519a-d1c2-4733-9182-9cdedf40f833-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.617750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.617593 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.618767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.618743 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:01.618867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.618845 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:01.618910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.618873 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:01.618994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.618980 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh" pod "ed15519a-d1c2-4733-9182-9cdedf40f833" (UID: "ed15519a-d1c2-4733-9182-9cdedf40f833"). InnerVolumeSpecName "kube-api-access-r67mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:01.718613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.718539 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.718613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.718567 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.718613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.718579 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15519a-d1c2-4733-9182-9cdedf40f833-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.718613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.718592 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67mh\" (UniqueName: \"kubernetes.io/projected/ed15519a-d1c2-4733-9182-9cdedf40f833-kube-api-access-r67mh\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:01.728528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.728501 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" Nov 01 22:06:01.728666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.728505 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz" event={"ID":"5049bef6-c09d-402e-a301-7c665a20451e","Type":"ContainerDied","Data":"461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2"} Nov 01 22:06:01.728666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.728628 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461ad10cb86309f68de254159cd8b4c777e1ce9bd8f3cfaedafc30eab3b5e4a2" Nov 01 22:06:01.730701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.730677 5036 generic.go:356] "Generic (PLEG): container finished" podID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerID="af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b" exitCode=0 Nov 01 22:06:01.730781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.730718 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerDied","Data":"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b"} Nov 01 22:06:01.730781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.730765 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" event={"ID":"ed15519a-d1c2-4733-9182-9cdedf40f833","Type":"ContainerDied","Data":"dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c"} Nov 01 22:06:01.730880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.730794 5036 scope.go:122] "RemoveContainer" containerID="af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b" Nov 01 22:06:01.730880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.730804 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls" Nov 01 22:06:01.752679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.752658 5036 scope.go:122] "RemoveContainer" containerID="3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.765633 5036 scope.go:122] "RemoveContainer" containerID="af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:01.766032 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b\": container with ID starting with af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b not found: ID does not exist" containerID="af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.766074 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b"} err="failed to get container status \"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b\": rpc error: code = NotFound desc = could not find container \"af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b\": container with ID starting with af28ac708ad49c29b5caff3e6df10bd8df47411b1f40f0fa472f12c88eb6773b not found: ID does not exist" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.766097 5036 scope.go:122] "RemoveContainer" containerID="3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:01.772529 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef\": container with ID starting with 3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef not found: ID does not exist" containerID="3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef" Nov 01 22:06:01.774293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.772561 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef"} err="failed to get container status \"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef\": rpc error: code = NotFound desc = could not find container \"3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef\": container with ID starting with 3e9c644a2bc54b1d729047af5f8418689ae61cdfee24d98d08a13768b9c817ef not found: ID does not exist" Nov 01 22:06:01.783681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.783654 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:06:01.791293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.791263 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d506c6-openshift-default-55bb48b9b5-8k5ls"] Nov 01 22:06:01.803723 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:01.803701 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded15519a_d1c2_4733_9182_9cdedf40f833.slice/crio-dce65eda222e7b8b6f25572d37a8a215e42f0522c467fffb2da4270d5551ff6c\": RecentStats: unable to find data in memory cache]" Nov 01 22:06:01.856936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:01.856908 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:02.441294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:02.441261 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" path="/var/lib/kubelet/pods/ed15519a-d1c2-4733-9182-9cdedf40f833/volumes" Nov 01 22:06:02.441911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:02.441886 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e" path="/var/lib/kubelet/pods/f2ae8d5f-4722-4624-94cc-7d0c4e1f1d9e/volumes" Nov 01 22:06:02.643659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:02.643619 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:03.066465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:03.058847 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx"] Nov 01 22:06:03.079507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:03.077593 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ee0976-4q6sx"] Nov 01 22:06:03.258982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:03.258940 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:03.440531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:03.440382 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:04.234418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:04.233490 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:04.415298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:04.415265 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:04.428543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:04.428520 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:04.442735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:04.442704 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84651b4-5a93-4c47-8677-034aed60f775" path="/var/lib/kubelet/pods/f84651b4-5a93-4c47-8677-034aed60f775/volumes" Nov 01 22:06:04.825467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:04.825210 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:05.385953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:05.385911 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:05.580619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:05.580583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:06.215292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:06.215254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:06.656926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:06.656890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:06.787986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:06.787951 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:07.240026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:07.239941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:07.945475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:07.945430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:08.171093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:08.171050 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:08.297035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:08.296984 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:08.487999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:08.487957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:09.369387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.369352 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc"] Nov 01 22:06:09.370365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370319 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.370365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370363 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.370622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370426 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.370622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370437 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.370622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370539 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5049bef6-c09d-402e-a301-7c665a20451e" containerName="revoke-keys" Nov 01 22:06:09.370622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370546 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5049bef6-c09d-402e-a301-7c665a20451e" containerName="revoke-keys" Nov 01 22:06:09.370787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370691 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5049bef6-c09d-402e-a301-7c665a20451e" containerName="revoke-keys" Nov 01 22:06:09.370787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370712 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.370787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.370727 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed15519a-d1c2-4733-9182-9cdedf40f833" containerName="istio-proxy" Nov 01 22:06:09.376070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.376045 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.385011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.380242 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:06:09.385011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.380541 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:06:09.387152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.387121 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc"] Nov 01 22:06:09.449606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.449564 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:06:09.449606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.449606 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:06:09.477611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.477578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:09.512311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.512264 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8hp\" (UniqueName: \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.512601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.512458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.512949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.512919 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.614561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.614523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.614751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.614703 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8hp\" (UniqueName: \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.614821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.614805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.615498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.615475 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.617056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.617026 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.627554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.627483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8hp\" (UniqueName: \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\") pod \"maas-api-revoke-keys-e2e-api-b-0390e1-plknc\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.666774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.666744 5036 scope.go:122] "RemoveContainer" containerID="5c0e96d6fcf5f016e777745224fc230936b7474a88ae7d752ce84424d56d260f" Nov 01 22:06:09.679343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.679321 5036 scope.go:122] "RemoveContainer" containerID="86eac198a903c7f31003510fae1ce7c2b76612d5486ae75fecdc3f0c91d9d9e1" Nov 01 22:06:09.691101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.691076 5036 scope.go:122] "RemoveContainer" containerID="2e2c7c43c3409e789fd4a151f9eb7904396df962a1984b8aa244627a9308861b" Nov 01 22:06:09.703428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.703406 5036 scope.go:122] "RemoveContainer" containerID="6ad7b2f8f6534b059704df9934137e8b5b08faf37fb48dbb88519b81b43b3bf5" Nov 01 22:06:09.715953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.715933 5036 scope.go:122] "RemoveContainer" containerID="29bcd317f8f604c4fbb8388ceb3915f5b36afdfea7cffdf18d35b815bed43e83" Nov 01 22:06:09.719125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.719104 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:09.727338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.727318 5036 scope.go:122] "RemoveContainer" containerID="6257654dbc72a638f30f59009aacdf5c92fbcba845ee629fe1e4abadc79705c6" Nov 01 22:06:09.742071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.742052 5036 scope.go:122] "RemoveContainer" containerID="27edddef98d6dbf1081f2476e2a5c954f51ea34e34580cbfbff20b7b43d9f545" Nov 01 22:06:09.947394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:09.947361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:10.039836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.031853 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc"] Nov 01 22:06:10.296217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.296182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:10.452156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.452120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:10.471489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.469386 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:10.476164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.474400 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.503738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.503498 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:10.538789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.538755 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.538911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.538834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.538973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.538938 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtmw\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539001 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539165 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539282 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539322 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.539413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.539387 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.552757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.552686 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:10.552757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.552744 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:10.552948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.552848 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.640854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.640823 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.640881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.640971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtmw\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641141 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641212 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641258 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641288 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641310 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641312 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641522 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641568 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641706 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641750 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641789 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.641927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641826 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.641927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.641878 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94f8h\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.642212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.642094 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.642212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.642161 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.642304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.642229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.643885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.643866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.644075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.644059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.653204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.653183 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtmw\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.653296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.653186 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-5jthv\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.743511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743475 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743546 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743562 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743762 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743798 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743841 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94f8h\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.743952 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.744007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.744031 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.744267 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.744886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.744716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.746798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.746772 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.747264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.747245 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.758346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.758319 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.758630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.758609 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94f8h\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:10.802943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.802861 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:10.890466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:10.889837 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:11.081820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.039714 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" event={"ID":"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67","Type":"ContainerStarted","Data":"30f350bf8feac5210c748fe192f110875d4e81dd24dabe31f8eb6033ff610248"} Nov 01 22:06:11.081820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.039781 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" event={"ID":"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67","Type":"ContainerStarted","Data":"aed20eeec9dffe22a10220cf5d84ea5db847273c278e7b6de0d6d35d9cdbcba7"} Nov 01 22:06:11.081820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.066676 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" podStartSLOduration=2.066656565 podStartE2EDuration="2.066656565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:11.066340885 +0000 UTC m=+1324.759321371" watchObservedRunningTime="2026-11-01 22:06:11.066656565 +0000 UTC m=+1324.759637054" Nov 01 22:06:11.117703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.117524 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:11.316491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.316434 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:11.335987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.335927 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.336103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.336036 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.336103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.336077 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.434056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.434029 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:11.435617 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:06:11.435594 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb59d8b_8b66_4995_8474_83cc680a6027.slice/crio-7fb9937f3ad5b26e6010cda94cfd9996f617dac3a36e9c002132e7a1a7a409bf WatchSource:0}: Error finding container 7fb9937f3ad5b26e6010cda94cfd9996f617dac3a36e9c002132e7a1a7a409bf: Status 404 returned error can't find the container with id 7fb9937f3ad5b26e6010cda94cfd9996f617dac3a36e9c002132e7a1a7a409bf Nov 01 22:06:11.439639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.439607 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.439743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.439682 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.439743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.439726 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:11.471537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.471506 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:11.514119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.512773 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:06:11.713471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.690004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:11.772474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.771501 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:06:11.772474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.771637 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.857903 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvjx\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858002 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858070 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858119 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858417 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858486 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.862466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.858616 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.879915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.878762 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d0a5de-openshift-default-dockercfg-28rxc\"" Nov 01 22:06:11.879915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.879216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:11.960335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.960247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.960619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.960595 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.960895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.960874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.961222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.961102 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.961415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.961396 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.974464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.973607 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.974464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.973775 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.974464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.973987 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvjx\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.974464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.974090 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:11.975021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.974995 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.012159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.977154 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.012159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.977407 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.012159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:11.978276 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.012159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.006521 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.012159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.012073 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.014928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.014838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvjx\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.015703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.015633 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.044463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.024024 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\") pod \"e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.165636 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" event={"ID":"2dcdba77-3e98-432a-a76c-42faedc3b9ec","Type":"ContainerStarted","Data":"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a"} Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.165730 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" event={"ID":"2dcdba77-3e98-432a-a76c-42faedc3b9ec","Type":"ContainerStarted","Data":"cdae580eb58986d6a6775a3ea3ab796937bb11d8579ad7c95db65fa0191a25c6"} Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.169996 5036 generic.go:356] "Generic (PLEG): container finished" podID="ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" containerID="30f350bf8feac5210c748fe192f110875d4e81dd24dabe31f8eb6033ff610248" exitCode=0 Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.170218 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" event={"ID":"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67","Type":"ContainerDied","Data":"30f350bf8feac5210c748fe192f110875d4e81dd24dabe31f8eb6033ff610248"} Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.173940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" event={"ID":"2fb59d8b-8b66-4995-8474-83cc680a6027","Type":"ContainerStarted","Data":"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36"} Nov 01 22:06:12.175469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.174846 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" event={"ID":"2fb59d8b-8b66-4995-8474-83cc680a6027","Type":"ContainerStarted","Data":"7fb9937f3ad5b26e6010cda94cfd9996f617dac3a36e9c002132e7a1a7a409bf"} Nov 01 22:06:12.238852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.238730 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" podStartSLOduration=2.23871461 podStartE2EDuration="2.23871461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:12.206886437 +0000 UTC m=+1325.899866924" watchObservedRunningTime="2026-11-01 22:06:12.23871461 +0000 UTC m=+1325.931695095" Nov 01 22:06:12.240019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.239963 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" podStartSLOduration=2.239952748 podStartE2EDuration="2.239952748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:12.229680234 +0000 UTC m=+1325.922660743" watchObservedRunningTime="2026-11-01 22:06:12.239952748 +0000 UTC m=+1325.932933234" Nov 01 22:06:12.242152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.242116 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:12.445904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.445713 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:12.649981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.649924 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:12.650404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.650038 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:12.650404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.650101 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:12.657430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.657400 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:06:12.806188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.805717 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:12.813877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.812175 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:06:12.813877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.812225 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 22:06:12.893950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.892583 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:12.893950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.892727 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 22:06:12.893950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.892766 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 22:06:12.972704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:12.972591 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:13.188415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.187594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerStarted","Data":"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6"} Nov 01 22:06:13.188415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.187676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerStarted","Data":"e9156e6dd6fc496755135260a1ec128b5b90e72698b980743b0ff5babcacbba8"} Nov 01 22:06:13.200170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.200100 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:13.220676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.220620 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podStartSLOduration=2.220602554 podStartE2EDuration="2.220602554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:13.219724001 +0000 UTC m=+1326.912704490" watchObservedRunningTime="2026-11-01 22:06:13.220602554 +0000 UTC m=+1326.913583041" Nov 01 22:06:13.246127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.246085 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:13.253247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.253212 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:13.377115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.376525 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:13.416469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.413830 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:13.568474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.545041 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:13.568474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.546772 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:13.763141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.754580 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:06:13.763141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.754872 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" containerID="cri-o://3fcf7c8628a08b535ae105b2810795f8b7465fc9229bd9a24ebb64c64f68136a" gracePeriod=30 Nov 01 22:06:13.928518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.927625 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 22:06:13.928518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.927689 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 22:06:13.971476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.932795 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:06:13.971476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:13.932878 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 22:06:14.159518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.157908 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:14.239416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.239324 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\") pod \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " Nov 01 22:06:14.240634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.240324 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca" pod "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" (UID: "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:14.240951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.240931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\" (UniqueName: \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\") pod \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " Nov 01 22:06:14.241843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.241437 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\") pod \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\" (UID: \"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67\") " Nov 01 22:06:14.242833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.242806 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.251992 5036 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" secret="" err="secret \"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\" not found" Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.252500 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.252654 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc" event={"ID":"ea312c5a-23d0-4a0e-8617-a08ac6ed3a67","Type":"ContainerDied","Data":"aed20eeec9dffe22a10220cf5d84ea5db847273c278e7b6de0d6d35d9cdbcba7"} Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.252692 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed20eeec9dffe22a10220cf5d84ea5db847273c278e7b6de0d6d35d9cdbcba7" Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.254769 5036 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" secret="" err="secret \"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\" not found" Nov 01 22:06:14.255542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.255315 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:14.257999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.257726 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:06:14.268722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.268678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca" pod "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" (UID: "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:14.269050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.269027 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp" pod "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" (UID: "ea312c5a-23d0-4a0e-8617-a08ac6ed3a67"). InnerVolumeSpecName "kube-api-access-sq8hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:14.410209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.410174 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8hp\" (UniqueName: \"kubernetes.io/projected/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-kube-api-access-sq8hp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.410209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.410209 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.636154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.636120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:14.817054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.817011 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:14.819271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.818620 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:06:14.819271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.819026 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 22:06:14.892111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.891619 5036 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 22:06:14.892111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:14.891675 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 22:06:15.261186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:15.261107 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" containerID="cri-o://3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36" gracePeriod=30 Nov 01 22:06:15.261428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:15.261216 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" containerID="cri-o://6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a" gracePeriod=30 Nov 01 22:06:15.573181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:15.573117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:15.696997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:15.696960 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:16.549389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:16.549355 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:17.304485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:17.303284 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:17.519040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:17.518027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:18.060574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:18.060529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:19.051738 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:19.051661 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:06:19.141507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.141381 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:19.376236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.376199 5036 generic.go:356] "Generic (PLEG): container finished" podID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerID="3fcf7c8628a08b535ae105b2810795f8b7465fc9229bd9a24ebb64c64f68136a" exitCode=0 Nov 01 22:06:19.380470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.376270 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerDied","Data":"3fcf7c8628a08b535ae105b2810795f8b7465fc9229bd9a24ebb64c64f68136a"} Nov 01 22:06:19.380470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.376534 5036 scope.go:122] "RemoveContainer" containerID="438831d60906510dea79d69725665b1c3e96f361180cc4999bef26de864f9e94" Nov 01 22:06:19.628470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.628312 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660405 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660564 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660582 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660609 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.660759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660748 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.668213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660802 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.668213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.660828 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\") pod \"6156be48-32e6-48dd-929a-9b00b1f27c9e\" (UID: \"6156be48-32e6-48dd-929a-9b00b1f27c9e\") " Nov 01 22:06:19.668213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.663945 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:19.668213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.666924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:19.668730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.668702 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:19.669043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.669021 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:19.669535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.669509 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:19.670242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.670046 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:19.671053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.671009 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:19.672995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.672965 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "kube-api-access-mvr8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:19.673965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.673941 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:19.675842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.675671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy" pod "6156be48-32e6-48dd-929a-9b00b1f27c9e" (UID: "6156be48-32e6-48dd-929a-9b00b1f27c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766726 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766759 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766775 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvr8x\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-kube-api-access-mvr8x\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766790 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766804 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766817 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6156be48-32e6-48dd-929a-9b00b1f27c9e-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766833 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766849 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6156be48-32e6-48dd-929a-9b00b1f27c9e-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.779082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.766861 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6156be48-32e6-48dd-929a-9b00b1f27c9e-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.963468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:19.934871 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:20.247469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.241065 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:20.420155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.420113 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" event={"ID":"6156be48-32e6-48dd-929a-9b00b1f27c9e","Type":"ContainerDied","Data":"d875d8b8a1d9d27616ce3a2d9a10f856be1ad69151698ea90dd5d99fddcc7c14"} Nov 01 22:06:20.420338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.420125 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4" Nov 01 22:06:20.420338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.420191 5036 scope.go:122] "RemoveContainer" containerID="3fcf7c8628a08b535ae105b2810795f8b7465fc9229bd9a24ebb64c64f68136a" Nov 01 22:06:20.475845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.475814 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:06:20.486317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.486284 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-dnvq4"] Nov 01 22:06:20.892756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:20.892709 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:21.572089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:21.572040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:22.246324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:22.246288 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:22.441804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:22.441770 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" path="/var/lib/kubelet/pods/6156be48-32e6-48dd-929a-9b00b1f27c9e/volumes" Nov 01 22:06:22.938505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:22.938463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:23.574010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:23.573970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:24.331426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:24.331394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:25.080768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:25.080723 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:25.632299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:25.632260 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:06:25.806287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:25.804376 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:25.897262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:25.895554 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:26.007471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:25.987593 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:26.265406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.265379 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:26.293143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.292884 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:26.295923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.295882 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.295942 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.295976 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296046 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296147 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296186 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296270 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296321 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.296368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.296349 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\") pod \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\" (UID: \"2dcdba77-3e98-432a-a76c-42faedc3b9ec\") " Nov 01 22:06:26.297318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.297291 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.297430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.297394 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:26.297576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.297514 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.297727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.297707 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.297920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.297889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.299570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.299542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.300039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.300010 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:26.300149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.300062 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "kube-api-access-zrtmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:26.303258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.303233 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo" pod "2dcdba77-3e98-432a-a76c-42faedc3b9ec" (UID: "2dcdba77-3e98-432a-a76c-42faedc3b9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:26.398104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398068 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398122 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398228 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398321 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398395 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398493 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.398660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398529 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398565 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398690 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\") pod \"2fb59d8b-8b66-4995-8474-83cc680a6027\" (UID: \"2fb59d8b-8b66-4995-8474-83cc680a6027\") " Nov 01 22:06:26.398986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.398962 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399147 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399156 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399742 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399764 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399780 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399794 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399809 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399823 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399837 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399850 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb59d8b-8b66-4995-8474-83cc680a6027-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399864 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399880 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtmw\" (UniqueName: \"kubernetes.io/projected/2dcdba77-3e98-432a-a76c-42faedc3b9ec-kube-api-access-zrtmw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399895 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399908 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcdba77-3e98-432a-a76c-42faedc3b9ec-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399922 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.399966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.399936 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.400951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.400785 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "kube-api-access-94f8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:26.401098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.401073 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:26.401157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.401100 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:26.402005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.401975 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy" pod "2fb59d8b-8b66-4995-8474-83cc680a6027" (UID: "2fb59d8b-8b66-4995-8474-83cc680a6027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:26.473233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.473200 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerID="3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36" exitCode=0 Nov 01 22:06:26.473351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.473250 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" Nov 01 22:06:26.473351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.473294 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" event={"ID":"2fb59d8b-8b66-4995-8474-83cc680a6027","Type":"ContainerDied","Data":"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36"} Nov 01 22:06:26.473509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.473357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x" event={"ID":"2fb59d8b-8b66-4995-8474-83cc680a6027","Type":"ContainerDied","Data":"7fb9937f3ad5b26e6010cda94cfd9996f617dac3a36e9c002132e7a1a7a409bf"} Nov 01 22:06:26.473509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.473389 5036 scope.go:122] "RemoveContainer" containerID="3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36" Nov 01 22:06:26.475731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.475707 5036 generic.go:356] "Generic (PLEG): container finished" podID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerID="6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a" exitCode=0 Nov 01 22:06:26.475859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.475753 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" Nov 01 22:06:26.475859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.475827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" event={"ID":"2dcdba77-3e98-432a-a76c-42faedc3b9ec","Type":"ContainerDied","Data":"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a"} Nov 01 22:06:26.475994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.475908 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv" event={"ID":"2dcdba77-3e98-432a-a76c-42faedc3b9ec","Type":"ContainerDied","Data":"cdae580eb58986d6a6775a3ea3ab796937bb11d8579ad7c95db65fa0191a25c6"} Nov 01 22:06:26.485989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.485966 5036 scope.go:122] "RemoveContainer" containerID="3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36" Nov 01 22:06:26.486291 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:26.486269 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36\": container with ID starting with 3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36 not found: ID does not exist" containerID="3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36" Nov 01 22:06:26.486346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.486300 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36"} err="failed to get container status \"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36\": rpc error: code = NotFound desc = could not find container \"3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36\": container with ID starting with 3526a374b7c02a64564e67186b1f63079738e55e3e5eac8c69d04d4586504d36 not found: ID does not exist" Nov 01 22:06:26.486346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.486317 5036 scope.go:122] "RemoveContainer" containerID="6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a" Nov 01 22:06:26.496933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.496909 5036 scope.go:122] "RemoveContainer" containerID="6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a" Nov 01 22:06:26.497232 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:26.497207 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a\": container with ID starting with 6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a not found: ID does not exist" containerID="6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a" Nov 01 22:06:26.497298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.497244 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a"} err="failed to get container status \"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a\": rpc error: code = NotFound desc = could not find container \"6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a\": container with ID starting with 6da9e6ec84ab56595fd194f8cb5869d7d44f7ccaaff938df04daa7efdf78eb7a not found: ID does not exist" Nov 01 22:06:26.501296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.501219 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94f8h\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-kube-api-access-94f8h\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.501296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.501247 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.501296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.501263 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.501296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.501277 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb59d8b-8b66-4995-8474-83cc680a6027-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:26.525856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.525831 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:26.551621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.551601 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-5jthv"] Nov 01 22:06:26.565531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.565505 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:26.569131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.569114 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-8mc6x"] Nov 01 22:06:26.901339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:26.901308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:28.499819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.499777 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:28.517810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.517745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:28.529494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.522720 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" path="/var/lib/kubelet/pods/2dcdba77-3e98-432a-a76c-42faedc3b9ec/volumes" Nov 01 22:06:28.529494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.523564 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" path="/var/lib/kubelet/pods/2fb59d8b-8b66-4995-8474-83cc680a6027/volumes" Nov 01 22:06:28.586348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.586299 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:28.666658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:28.666596 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.054480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.052542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.382722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.382617 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.428284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.428244 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.499639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.472321 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:29.500672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.499944 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.535823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.518200 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:29.535823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.518746 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:29.572789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.537292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:29.690521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.671986 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:29.690521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.672557 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:29.694012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.691561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.813861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.799925 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:06:29.943594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.814137 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:30.017537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.814175 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:06:30.017537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:29.855962 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.059938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.052349 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.059938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.058972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.085691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.085660 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:30.587468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.542887 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.805805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.805747 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.816338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.816293 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.924765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.924656 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.926661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.926612 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.957631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.957589 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:06:30.968647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.958274 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.013841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:30.981312 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:06:31.117486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.106485 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.214522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.196341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:31.240066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.240033 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:31.257763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.257731 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:31.304914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.304884 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:31.305167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305151 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:06:31.305342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305329 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:31.305505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305490 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:31.305686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305671 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:31.305823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305811 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:06:31.319467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.305942 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:06:31.320018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.319995 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:06:31.367474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.357732 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:31.559522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.553709 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz"] Nov 01 22:06:31.579024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.559783 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.632364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.632324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:31.726681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.689533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:31.727022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.696209 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:06:31.757287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.756856 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.761606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.761537 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.820247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.820133 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.919199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.912149 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-004180-mwfhz"] Nov 01 22:06:31.936983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.936926 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.937727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.937599 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:31.970550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.970517 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:31.971917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:31.971884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.003131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.003098 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:06:32.024786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.024753 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:32.025687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.025651 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.039958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.029421 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:32.110629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.110531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.188276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.188239 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:06:32.188596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.188566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.357504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.350554 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:06:32.357504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.350621 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.453854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.453765 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:32.634474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.630719 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5049bef6-c09d-402e-a301-7c665a20451e" path="/var/lib/kubelet/pods/5049bef6-c09d-402e-a301-7c665a20451e/volumes" Nov 01 22:06:32.634474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.631226 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:32.634474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.631639 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:32.762429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.762003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:32.775286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.769071 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.775929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.775904 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:06:32.978495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:32.971182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:33.048491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.012874 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.048491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.042722 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.122779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.122727 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.502522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.483592 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.610635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.610591 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.709469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.693366 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:33.736610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.736549 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.809474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.799689 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:33.844471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:33.831616 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.005618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.005564 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.062523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.046674 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.062523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.046625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.062523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.047847 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:06:34.066226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.066191 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.072176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.072140 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:34.188469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.140730 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:06:34.189295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.159697 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.343519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.321627 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.486891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.486847 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:06:34.588699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.588649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.608364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.608092 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:34.711471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.703827 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:35.216256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.216211 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:35.217173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.217140 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:06:35.222562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.222526 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.223026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.222991 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:35.227844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.214596 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.230121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.228007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:35.230121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.868545 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.230121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.868820 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.230121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.938952 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.235319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:34.853534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:35.267707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.267533 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:35.307574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.307528 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 01 22:06:35.320508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.320479 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:35.322558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.322534 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:35.326622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.326601 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:35.326811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.326797 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:35.326935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.326922 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:35.327072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.327060 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:06:35.431817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.429573 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.446516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.432628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:35.472993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.472912 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:06:35.502992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.502945 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.677477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.653221 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.677477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.654104 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.709541 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn"] Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710551 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" containerName="revoke-keys" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710575 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" containerName="revoke-keys" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710619 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710627 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710636 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710645 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710669 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710678 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710694 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710702 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710892 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710906 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcdba77-3e98-432a-a76c-42faedc3b9ec" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710919 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb59d8b-8b66-4995-8474-83cc680a6027" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710931 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6156be48-32e6-48dd-929a-9b00b1f27c9e" containerName="istio-proxy" Nov 01 22:06:35.711529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.710949 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" containerName="revoke-keys" Nov 01 22:06:35.751513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.737714 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:35.751513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.737934 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:35.760478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.751946 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn"] Nov 01 22:06:35.787865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.761344 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:06:35.787865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.761470 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:06:35.787865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.761530 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:06:35.813028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.812993 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:35.813885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.813753 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:35.846859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.844237 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:06:35.884673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.884622 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:06:35.927633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.922417 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:35.927633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.922542 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:35.927633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.922753 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79tq\" (UniqueName: \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:35.937015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:35.928029 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.023912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.023876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.025505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.025480 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.026187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.025284 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.026523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.026500 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79tq\" (UniqueName: \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.052836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.052765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.080266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.080231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79tq\" (UniqueName: \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\") pod \"maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.091029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.091001 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:36.228900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.197757 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:36.229413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.229396 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:36.229597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.225843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.229779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.228579 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.230105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.228619 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.242470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.228679 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.242470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.224115 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:36.242470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.230647 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:36.253578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.252757 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:36.530573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.530529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:06:36.948781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:36.948729 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:37.001641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.001602 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:37.038715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.038672 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.106979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.106922 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.139985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.139943 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.173764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.173715 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.191493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.190771 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:37.240002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.238653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.240002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.239031 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:37.310494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.300784 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.323471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.318287 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.384520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.335343 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.466104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.465050 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:37.533467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.533403 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.876023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.875935 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:06:37.878506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.878483 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:06:37.891545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.891511 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:37.952758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.952644 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:37.985418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:37.985392 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:38.136650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.113725 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.136650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.114527 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:38.136650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.120469 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.196018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.159893 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.196018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.159966 5036 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:06:38.196018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.160762 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:06:38.196018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.182999 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.196018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.183567 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:38.271529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.254907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.278058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.277021 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:06:38.325657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.324895 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.351431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.347795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:38.351431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.349087 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:38.369999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.369975 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:38.540934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.535831 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:38.558133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.558100 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:06:38.653483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.651605 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:38.654488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.654410 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:38.763823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:38.762198 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn"] Nov 01 22:06:39.172570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.172486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" event={"ID":"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54","Type":"ContainerStarted","Data":"c885acad555e291973d617a2c69edfa9d749b190ab546b1af3bf2eda72620d6d"} Nov 01 22:06:39.341308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.341260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:39.632559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.632515 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:39.834116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.834046 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:39.931048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.927895 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:06:39.931048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.928826 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" containerID="cri-o://41515f3a543973c4695ee1967d9eeb6fb2ba56b5d6554817f3f571d508cb939f" gracePeriod=30 Nov 01 22:06:39.968890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:39.968841 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:40.210114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:40.179268 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:40.210114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:40.194291 5036 generic.go:356] "Generic (PLEG): container finished" podID="0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" containerID="c5ff86b0bfafa9d14f367f35dc437878f06d88ea5ece787c2ed3c473e77ceaf6" exitCode=0 Nov 01 22:06:40.210114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:40.194559 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" event={"ID":"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54","Type":"ContainerDied","Data":"c5ff86b0bfafa9d14f367f35dc437878f06d88ea5ece787c2ed3c473e77ceaf6"} Nov 01 22:06:40.711995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:40.711958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:40.916428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:40.916389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:41.289542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.282906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:41.860166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.859325 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:41.890466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.890358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:41.916428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.915951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\") pod \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " Nov 01 22:06:41.916428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.916069 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\" (UniqueName: \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\") pod \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " Nov 01 22:06:41.916428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.916152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\") pod \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\" (UID: \"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54\") " Nov 01 22:06:41.918181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.917861 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca" pod "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" (UID: "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:41.938696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.929412 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq" pod "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" (UID: "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54"). InnerVolumeSpecName "kube-api-access-t79tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:41.938696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:41.931577 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca" pod "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" (UID: "0a49f3ff-9d30-44c7-bb66-6f7696fa8d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:42.056483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.048425 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:42.056483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.048473 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79tq\" (UniqueName: \"kubernetes.io/projected/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-kube-api-access-t79tq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:42.056483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.048487 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:42.056483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.049195 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:42.170188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.170066 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:06:42.190970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.171542 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" containerName="revoke-keys" Nov 01 22:06:42.190970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.171572 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" containerName="revoke-keys" Nov 01 22:06:42.190970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.171893 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" containerName="revoke-keys" Nov 01 22:06:42.190970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.179719 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.190970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.187055 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:06:42.257837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257598 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.257837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257688 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.257837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257736 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.257837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257796 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.257837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257831 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.258244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257868 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.258244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257941 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.258244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.257987 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.258244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.258025 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.327466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.327284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" event={"ID":"0a49f3ff-9d30-44c7-bb66-6f7696fa8d54","Type":"ContainerDied","Data":"c885acad555e291973d617a2c69edfa9d749b190ab546b1af3bf2eda72620d6d"} Nov 01 22:06:42.327466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.327345 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c885acad555e291973d617a2c69edfa9d749b190ab546b1af3bf2eda72620d6d" Nov 01 22:06:42.327466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.327414 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359688 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359758 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359866 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359907 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.359999 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.360032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.360220 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.361116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.361541 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.361819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.361789 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.363788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.362566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.363788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.363162 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.364896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.364862 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.366774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.366751 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.380988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.380936 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.381996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.381971 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.535518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.529146 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:42.787550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.787110 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:06:42.787550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.787174 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:42.867986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.867821 5036 patch_prober.go:29] interesting pod/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:06:42.867986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.867881 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:06:42.908345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:42.908303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:43.017653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.004076 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:06:43.027206 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:06:43.027172 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050cdada_3493_47ec_9836_3d51ac6373e0.slice/crio-921d72ffaf3da920b81235335b9b693199c4ef8d2c99f70aecc8115ea177dd7c WatchSource:0}: Error finding container 921d72ffaf3da920b81235335b9b693199c4ef8d2c99f70aecc8115ea177dd7c: Status 404 returned error can't find the container with id 921d72ffaf3da920b81235335b9b693199c4ef8d2c99f70aecc8115ea177dd7c Nov 01 22:06:43.031489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.031433 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:43.031591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.031541 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:43.031591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.031583 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:43.357774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.357648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" event={"ID":"050cdada-3493-47ec-9836-3d51ac6373e0","Type":"ContainerStarted","Data":"5386de844a8fb690ad35676f2cc8af3c9d9dad1b22f9a96c07c23c5e625c8ef7"} Nov 01 22:06:43.357774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.357720 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" event={"ID":"050cdada-3493-47ec-9836-3d51ac6373e0","Type":"ContainerStarted","Data":"921d72ffaf3da920b81235335b9b693199c4ef8d2c99f70aecc8115ea177dd7c"} Nov 01 22:06:43.399655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.399125 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podStartSLOduration=1.399104221 podStartE2EDuration="1.399104221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:43.388105129 +0000 UTC m=+1357.081085616" watchObservedRunningTime="2026-11-01 22:06:43.399104221 +0000 UTC m=+1357.092084708" Nov 01 22:06:43.530244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.530141 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:43.957884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:43.957836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:44.456509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:44.455935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:44.541593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:44.541543 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:44.541787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:44.541618 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:44.651467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:44.617084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:45.022742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.022597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:45.271795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.271623 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:45.470608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.470568 5036 generic.go:356] "Generic (PLEG): container finished" podID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerID="41515f3a543973c4695ee1967d9eeb6fb2ba56b5d6554817f3f571d508cb939f" exitCode=0 Nov 01 22:06:45.471214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.471187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerDied","Data":"41515f3a543973c4695ee1967d9eeb6fb2ba56b5d6554817f3f571d508cb939f"} Nov 01 22:06:45.471427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.471408 5036 scope.go:122] "RemoveContainer" containerID="650ae3df6ea3cfeca4dad308e68a67d5fe43bfecd75cf543c9b8c64695813132" Nov 01 22:06:45.531015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.530936 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:45.531189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.531018 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:45.553799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.553746 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:45.670582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.670544 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:06:45.746635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.746635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.746635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746353 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.746635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746422 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.746635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746517 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.747071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.747071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746723 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.747071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746771 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.747071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.746848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\") pod \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\" (UID: \"d8d056d7-ba24-40f6-8979-fe61cb7b7da8\") " Nov 01 22:06:45.747999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.747970 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:45.748223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.748199 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:45.748306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.748221 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:45.748373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.748299 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:45.748658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.748623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:45.763812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.763777 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:45.763940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.763877 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:45.764050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.764020 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:45.766204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.766174 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr" pod "d8d056d7-ba24-40f6-8979-fe61cb7b7da8" (UID: "d8d056d7-ba24-40f6-8979-fe61cb7b7da8"). InnerVolumeSpecName "kube-api-access-qfbgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849433 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849506 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849524 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849540 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849561 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849583 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849600 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849616 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.061089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:45.849631 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbgr\" (UniqueName: \"kubernetes.io/projected/d8d056d7-ba24-40f6-8979-fe61cb7b7da8-kube-api-access-qfbgr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:46.441355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.316091 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.454370 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.515273 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.524028 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k" event={"ID":"d8d056d7-ba24-40f6-8979-fe61cb7b7da8","Type":"ContainerDied","Data":"0eab5a4d4517bb90602c284df4b5938d0dc329a9356d8f2781655ac78c09065f"} Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.524097 5036 scope.go:122] "RemoveContainer" containerID="41515f3a543973c4695ee1967d9eeb6fb2ba56b5d6554817f3f571d508cb939f" Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.553259 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:46.586185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.553313 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:46.732641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.731740 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:06:46.770371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.770335 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-004180-openshift-default-7dfb78476f-25d2k"] Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.859044 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860193 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860216 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860239 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860247 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860492 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.862304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.860511 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" containerName="istio-proxy" Nov 01 22:06:46.867849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.866842 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:46.867849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:46.867589 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:06:47.033036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.032997 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033132 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033176 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033201 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2bz\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033292 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033499 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.033714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.033560 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.134763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.134728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.134879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.134863 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.134923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.134900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.134959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.134937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.134962 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2bz\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135073 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135220 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135276 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135301 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135605 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135731 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.135797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.135748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.137635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.137612 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.137862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.137847 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.146847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.146825 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.147105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.147086 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2bz\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.209466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.208811 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:47.362826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.361977 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:47.530681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.530645 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:47.530893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.530708 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.533271 5036 generic.go:356] "Generic (PLEG): container finished" podID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerID="7e2eb5fc5e0891f6722c72c15186e591c25d7645b76228f3bd89e5a9ca19b24f" exitCode=0 Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.533347 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerDied","Data":"7e2eb5fc5e0891f6722c72c15186e591c25d7645b76228f3bd89e5a9ca19b24f"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.533920 5036 scope.go:122] "RemoveContainer" containerID="7e2eb5fc5e0891f6722c72c15186e591c25d7645b76228f3bd89e5a9ca19b24f" Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.536491 5036 generic.go:356] "Generic (PLEG): container finished" podID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerID="85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2" exitCode=0 Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.536751 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerDied","Data":"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.537260 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.537386 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.537474 5036 scope.go:122] "RemoveContainer" containerID="85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2" Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.537472 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.540528 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.540599 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.542496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.540664 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.560603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.560564 5036 generic.go:356] "Generic (PLEG): container finished" podID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerID="10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123" exitCode=0 Nov 01 22:06:47.560895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.560864 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerDied","Data":"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123"} Nov 01 22:06:47.561664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.561643 5036 scope.go:122] "RemoveContainer" containerID="10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123" Nov 01 22:06:47.570264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.569776 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.570264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.569870 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.570264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.569923 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.625313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.624566 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:06:47.625313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.624935 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:06:47.636278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.636235 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.636403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.636347 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:47.636539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:47.636438 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:06:48.449399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.449354 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d056d7-ba24-40f6-8979-fe61cb7b7da8" path="/var/lib/kubelet/pods/d8d056d7-ba24-40f6-8979-fe61cb7b7da8/volumes" Nov 01 22:06:48.535298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.534861 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:48.535298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.534937 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:48.572522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.572494 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" event={"ID":"5ee2ae42-241a-44a2-9332-72502fd4c5ff","Type":"ContainerStarted","Data":"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff"} Nov 01 22:06:48.572747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.572731 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" event={"ID":"5ee2ae42-241a-44a2-9332-72502fd4c5ff","Type":"ContainerStarted","Data":"89a9cdb37d2264eb8ec267e5b9a78c754605fb86cecb0d64d8fa2a5dcfbe55b2"} Nov 01 22:06:48.575274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.575240 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerStarted","Data":"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc"} Nov 01 22:06:48.580470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.579156 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerStarted","Data":"e3de9af898709b87e0a01f9f22966c469f36b97dff0022215b123bc858b27a96"} Nov 01 22:06:48.592466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.582838 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerStarted","Data":"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c"} Nov 01 22:06:48.611468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.611198 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podStartSLOduration=2.611178451 podStartE2EDuration="2.611178451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:48.609506384 +0000 UTC m=+1362.302486873" watchObservedRunningTime="2026-11-01 22:06:48.611178451 +0000 UTC m=+1362.304158936" Nov 01 22:06:48.725071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.724950 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:06:48.726122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.726099 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:48.726523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.726496 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:48.806468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.805493 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 22:06:48.806468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.805779 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" containerID="cri-o://297d33e9081a19351fdc600901a0f74cb284b12382b875771ef27dae8050d094" gracePeriod=30 Nov 01 22:06:48.863469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.862964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:48.992744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:48.992590 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:06:49.017683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.016579 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.017683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.016649 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:49.332670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.304810 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:06:49.332670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.324078 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:06:49.353501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.333584 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.353501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.333642 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:49.353501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.333939 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.353501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.333974 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:49.394562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.382034 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:06:49.410483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.400280 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 22:06:49.410483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.400559 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" containerID="cri-o://e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318" gracePeriod=30 Nov 01 22:06:49.590558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.589924 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.590558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.589987 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:49.820411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.820358 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:49.825906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.825503 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.825906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:49.825577 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:50.079526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.079056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:50.087012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.086958 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.205067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.134747 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.205067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.137224 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.205067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.165031 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.205067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.165097 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:50.335535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.304203 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.394630 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.394706 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.394850 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.394875 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.397391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.398566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:50.437472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.404882 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.559986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.559946 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.805633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.805569 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:50.805909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.664995 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.806062 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:49.749307 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:06:50.806166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.630527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:06:50.807166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.807137 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:06:50.807693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.807668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:50.843467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.815718 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.843467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.838856 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.843467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.838931 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:50.843467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.841306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:50.843467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.841664 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:50.907257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:50.907223 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:51.047472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.035295 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.047472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.035359 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:51.137983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.137886 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.138830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.138787 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:51.166467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.141356 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:06:51.214102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.192803 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:06:51.214369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.214338 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:06:51.214920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.214896 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:06:51.244897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.223574 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.256469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.245942 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.288630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.288592 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.288938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.288901 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:51.355578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.355525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.417751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.368507 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.426467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.422503 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:51.472474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.445589 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.472474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.446520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.472474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.446594 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:51.502054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.502015 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:06:51.502323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.502305 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:06:51.512468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.502564 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:51.532963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.532936 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:06:51.539535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.539504 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.539772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.539741 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 01 22:06:51.622047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.622009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:51.736034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.735942 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.788475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.769910 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.869467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.853922 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:51.869467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.813857 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:06:51.869467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.813880 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:06:51.869467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.813825 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:51.890318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.847648 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.890925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.829541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:51.892339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.892313 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:51.892564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.892546 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:06:51.892726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.892712 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:06:51.892866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:51.892853 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:52.103476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.064129 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:52.138747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.138710 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.226603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.226534 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:52.230673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.230636 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:52.249504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.162668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:52.275469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.260187 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:52.275469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.261568 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:52.275469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.272522 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.275469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.272562 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:52.436993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.435793 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.436993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.435869 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:52.524470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.517835 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:52.524470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.518503 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:52.524470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.518983 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:52.756257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.755596 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:06:52.756257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.755662 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:06:52.800470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.756610 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:52.800470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.757474 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.800470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.757528 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:52.854196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.854126 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:52.858810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.858763 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:52.877846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.877118 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:06:52.944997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.942716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:52.944997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.943104 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:06:52.961378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.961328 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.977566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.977518 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:52.992706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.986738 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:53.001594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:52.996298 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:53.036312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.036275 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:53.056847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.056804 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:06:53.083715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.083670 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:53.086275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.086249 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:53.259593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.259548 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.317405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.259870 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:06:53.357468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.291537 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.383506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.301584 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:53.386742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.386707 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:53.387236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.383762 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:53.387410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.387389 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:53.624101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.520146 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:53.624831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.538568 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.625373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.538610 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.625854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.538631 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:06:53.697052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.538650 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:53.697360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.557749 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.742778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.742740 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:53.743060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.743030 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:53.860463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.856828 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.885531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.881563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.885531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.882050 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.944635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.944571 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:53.947613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.947573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:53.963237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.951253 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:53.963950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:53.963920 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:54.086722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.085541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:54.232975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.230882 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:54.232975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.230960 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:54.263069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.258994 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:54.292502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.289667 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:54.341854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.341811 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:54.342617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.341933 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:54.342848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.342825 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:54.364940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.342027 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 22:06:54.365180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.342003 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:54.385700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.385638 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:54.625584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.625517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:54.625986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.625951 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:54.752371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.751848 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:06:54.757218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.757185 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:54.855483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.854424 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Nov 01 22:06:54.877171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.856754 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:54.877171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.856805 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:54.879125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.879071 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:54.896616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.896573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:54.912470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.897392 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:06:55.000876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.912755 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:06:55.000876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:54.946348 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.196472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.192863 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:55.286465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.272961 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:55.286465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.273014 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:55.331603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.307213 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:55.331603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.307291 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.351683 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.372632 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.403162 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.408579 5036 generic.go:356] "Generic (PLEG): container finished" podID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerID="297d33e9081a19351fdc600901a0f74cb284b12382b875771ef27dae8050d094" exitCode=0 Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.408721 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" event={"ID":"e3c64037-3253-45bf-8499-8dda6bdb0437","Type":"ContainerDied","Data":"297d33e9081a19351fdc600901a0f74cb284b12382b875771ef27dae8050d094"} Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.410185 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 01 22:06:55.411691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.411042 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.435546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.435423 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.510595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.510561 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:55.510907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.510860 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:55.690218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.562860 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:55.691066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.676138 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 22:06:55.691390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.686185 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.691845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.686227 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.755471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.748257 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:55.755471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.748308 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:55.760609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.760540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.861707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.861663 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.896601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.896524 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:55.935502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.935470 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:55.960855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.960751 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:55.966156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.966021 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:55.966411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.966383 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:55.993068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.993038 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:55.993793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.993756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:56.061468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.048083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:56.061468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.048225 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:56.061468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.048284 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\") pod \"e3c64037-3253-45bf-8499-8dda6bdb0437\" (UID: \"e3c64037-3253-45bf-8499-8dda6bdb0437\") " Nov 01 22:06:56.108470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.936411 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:56.109000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.990651 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:56.109174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.109144 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:56.109305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.954821 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:56.110692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.956273 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:06:56.111842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.956229 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.990870 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.101876 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.102107 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.074693 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:55.956297 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.151102 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.151134 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.151149 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.151163 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c64037-3253-45bf-8499-8dda6bdb0437-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.156932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.151177 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.250695 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.250762 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.251314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.254072 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.280690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.306749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:56.359469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.324137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:56.414437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.406552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:56.453468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.415733 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:56.453468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.416066 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:06:56.453468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.417558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:56.454883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.454736 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.454883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.454782 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c64037-3253-45bf-8499-8dda6bdb0437-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.606666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.532817 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c" pod "e3c64037-3253-45bf-8499-8dda6bdb0437" (UID: "e3c64037-3253-45bf-8499-8dda6bdb0437"). InnerVolumeSpecName "kube-api-access-9h94c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:56.606666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.578787 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h94c\" (UniqueName: \"kubernetes.io/projected/e3c64037-3253-45bf-8499-8dda6bdb0437-kube-api-access-9h94c\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:56.773527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.692856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:56.773527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.719074 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:56.773527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.719170 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:06:56.824501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.802533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:06:56.869544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.863039 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:56.869544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.863099 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:56.931567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.920059 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:56.931567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.921211 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:56.931567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.921302 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:06:56.933030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.932420 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:56.933030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.932492 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.934132 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" event={"ID":"e3c64037-3253-45bf-8499-8dda6bdb0437","Type":"ContainerDied","Data":"b9946d2d493f1eafe378c633476836dbe7f059983dcbe477d82d7b102fdd656c"} Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.934205 5036 scope.go:122] "RemoveContainer" containerID="297d33e9081a19351fdc600901a0f74cb284b12382b875771ef27dae8050d094" Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.934427 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph" Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.944258 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.944304 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.944883 5036 generic.go:356] "Generic (PLEG): container finished" podID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerID="e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318" exitCode=0 Nov 01 22:06:56.955476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:56.945131 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" event={"ID":"b37a2f2d-e9f4-45c3-b081-e0213124dfac","Type":"ContainerDied","Data":"e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318"} Nov 01 22:06:57.091573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.073799 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:06:57.091573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.074964 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" Nov 01 22:06:57.091573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.074987 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" Nov 01 22:06:57.091573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.075178 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" containerName="istio-proxy" Nov 01 22:06:57.110838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.109501 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.111928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.111213 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.111928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.111293 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.120659 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.126963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.191427 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.194648 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.196846 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d8b5d-openshift-default-dockercfg-pxzwg\"" Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.215339 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:57.225535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.215387 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:57.318488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.314265 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.416806 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.418100 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.419547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.419789 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429085 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429126 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429171 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429219 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rz6\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429365 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429487 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.429663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.430569 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.431687 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:57.434558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.431723 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:57.503537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.494241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:57.543258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.542545 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 22:06:57.544321 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:06:57.544291 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37a2f2d_e9f4_45c3_b081_e0213124dfac.slice/crio-conmon-e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.544594 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.544667 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.544698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.544737 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rz6\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.544923 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.545048 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.545271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.545157 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.556913 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.557703 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.557968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.545249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.598195 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.599115 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.600577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.599313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.820536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.727888 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:57.820536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.777154 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rz6\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.873559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877481 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877543 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877845 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877897 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.877947 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.878040 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.878297 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.878561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\") pod \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\" (UID: \"b37a2f2d-e9f4-45c3-b081-e0213124dfac\") " Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.907513 5036 patch_prober.go:29] interesting pod/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.907632 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.960183 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.972347 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:58.005951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:57.972755 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.015479 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.018833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.020127 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.023618 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.023885 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.023940 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.024482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "kube-api-access-5mc7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.027251 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.027375 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.027436 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.032990 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.043784 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.047191 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.047503 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.113907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:58.171229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.130990 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.246350 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.246916 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.247019 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.253518 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.306229 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy" pod "b37a2f2d-e9f4-45c3-b081-e0213124dfac" (UID: "b37a2f2d-e9f4-45c3-b081-e0213124dfac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.307004 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:58.307508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.307056 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:58.359316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.334563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.395838 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.395922 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.440649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.441041 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.440607 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.513880 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.573434 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc7z\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-kube-api-access-5mc7z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.573502 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.573945 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.573975 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.574019 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.574042 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.574067 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.574100 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.574128 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a2f2d-e9f4-45c3-b081-e0213124dfac-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:58.659096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.579677 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.771616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.770076 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:06:58.884980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.884552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.939476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.936661 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cbpph"] Nov 01 22:06:58.939476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.936825 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:06:58.939476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.936925 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:58.939476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.936990 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:06:58.969470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.968878 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.969470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.969127 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:58.982473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.979014 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:58.982473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:58.979069 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:59.043329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.028084 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" Nov 01 22:06:59.043329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.028305 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp" event={"ID":"b37a2f2d-e9f4-45c3-b081-e0213124dfac","Type":"ContainerDied","Data":"9f456919df5aace9f67af5f3084e4e5fc9e20ebc59bda16d5320b3c75c6f4ae9"} Nov 01 22:06:59.043329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.028379 5036 scope.go:122] "RemoveContainer" containerID="e669617abf4f67d3530f82f0483f01c1467f9af3916cb97d6a2c511263aed318" Nov 01 22:06:59.052201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.051880 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:06:59.052201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.052023 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.052515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.052473 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.121009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.120167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:59.135751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.134602 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.135751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.135634 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:59.256932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.238649 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 22:06:59.256932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.238714 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 22:06:59.284516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.257292 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.448998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.381284 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:06:59.451772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.451739 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:06:59.460650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.442291 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:06:59.493719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.452522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.494061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.464191 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.529245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.528933 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:06:59.660810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.591601 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:06:59.660810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.622715 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 22:06:59.660810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.637202 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:06:59.660810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.641870 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:06:59.680822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.679374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:06:59.784517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.751715 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:06:59.784517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.761715 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-858wp"] Nov 01 22:06:59.907463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.905088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:59.970653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.962576 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:06:59.971199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.971156 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 22:06:59.979608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.979574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:59.995545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.995516 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:07:00.010479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.998760 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:00.034100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:06:59.999540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:00.034100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.012392 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:07:00.173256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.172992 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:00.240742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.232547 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:07:00.283684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.283433 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:07:00.299919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.296097 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:07:00.347145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.347111 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:07:00.370386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.370344 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 22:07:00.391666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.348700 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:07:00.404558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.404516 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:07:00.405472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.405415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:07:00.407178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.407154 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:07:00.411762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.411198 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:07:00.413373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.413312 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:00.413660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.413633 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:00.413828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.413804 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:00.429550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.419009 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:07:00.463023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.462993 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" path="/var/lib/kubelet/pods/b37a2f2d-e9f4-45c3-b081-e0213124dfac/volumes" Nov 01 22:07:00.466516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.466493 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c64037-3253-45bf-8499-8dda6bdb0437" path="/var/lib/kubelet/pods/e3c64037-3253-45bf-8499-8dda6bdb0437/volumes" Nov 01 22:07:00.712260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.712183 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:00.848832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.848798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:00.951374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.951337 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:07:00.952826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:00.952794 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:07:01.303932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.303893 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerStarted","Data":"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1"} Nov 01 22:07:01.304353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.303991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerStarted","Data":"c0ab7636c3508e6e6057d4fd81dafeea6665963fa578456cd436771150c82226"} Nov 01 22:07:01.321413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.321388 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:07:01.321712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.321685 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:07:01.322549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.322529 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:07:01.327799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.327759 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podStartSLOduration=5.327745131 podStartE2EDuration="5.327745131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:01.326652999 +0000 UTC m=+1375.019633487" watchObservedRunningTime="2026-11-01 22:07:01.327745131 +0000 UTC m=+1375.020725616" Nov 01 22:07:01.556276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.556191 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:01.771560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.771526 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:07:01.778058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:01.778036 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:07:02.113650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.113608 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:02.270491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.268475 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:02.402520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.402235 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:07:02.435414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.434603 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:07:02.436418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.436391 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:02.689514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.688085 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc"] Nov 01 22:07:02.693433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.693407 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0390e1-plknc"] Nov 01 22:07:02.945152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:02.945053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:03.063637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:03.063601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:03.196809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:03.196731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:04.092427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:04.092354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:04.303679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:04.303645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:04.433363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:04.433265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:04.462250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:04.462215 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea312c5a-23d0-4a0e-8617-a08ac6ed3a67" path="/var/lib/kubelet/pods/ea312c5a-23d0-4a0e-8617-a08ac6ed3a67/volumes" Nov 01 22:07:05.426809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:05.426772 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:05.553376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:05.553332 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:05.698518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:05.685520 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:06.489572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:06.489537 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:07.237662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:07.237622 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:07.944677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:07.944134 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:08.654655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:08.654622 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:08.792100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:08.792052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:09.065164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.065128 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:07:09.066278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.066246 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" Nov 01 22:07:09.066498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.066469 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" Nov 01 22:07:09.066780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.066761 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37a2f2d-e9f4-45c3-b081-e0213124dfac" containerName="istio-proxy" Nov 01 22:07:09.072684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.072661 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.093532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.093505 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0f9b7e-openshift-default-dockercfg-9mwpw\"" Nov 01 22:07:09.102124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.102096 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:07:09.144653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144624 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.144809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144678 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.144809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144752 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.144809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144774 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.144809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.145031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144813 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.145031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.145031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144900 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskmg\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.145031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.144987 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.159578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.159548 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:09.257485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.246138 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskmg\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.257485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.246273 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.257485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.246356 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.257485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.246408 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.257906 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.257969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.257998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.258037 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.258156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.258670 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.270481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.259493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.272752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.271778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.272752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.272030 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.272752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.272669 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.290526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.282963 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.290526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.288705 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.292532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.292483 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.326253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.326147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskmg\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\") pod \"e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.447778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.447748 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:09.671614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.663402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:09.874480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.874426 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:07:09.875108 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:09.875080 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedce6c5a_6cbf_4471_b32d_b98c8f6b33cc.slice/crio-ad80d5d290e535ee530de366eac3cb5331c3a659400778adf95164b5ec16aff1 WatchSource:0}: Error finding container ad80d5d290e535ee530de366eac3cb5331c3a659400778adf95164b5ec16aff1: Status 404 returned error can't find the container with id ad80d5d290e535ee530de366eac3cb5331c3a659400778adf95164b5ec16aff1 Nov 01 22:07:09.877930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.877886 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:09.878033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.877971 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:09.878033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.878013 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:09.939142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.939108 5036 scope.go:122] "RemoveContainer" containerID="b5a9aafbd3bed65aa6f5b50364d61da6af54fac551f4cf19f480a569e3d012e1" Nov 01 22:07:09.951461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.951417 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:09.956235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:09.956219 5036 scope.go:122] "RemoveContainer" containerID="cc68c0bf9bc8ea6f0eebc5f3bc90cf6ab74c911d12d0ac2ac77c37d51baa263e" Nov 01 22:07:10.144970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.143184 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:10.309477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.308702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:10.542499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.536697 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerStarted","Data":"91b7c7c3f5965fa88455ea30c62b88fa4e731f1611db7118f5a9860b483ef8bf"} Nov 01 22:07:10.542499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.536766 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerStarted","Data":"ad80d5d290e535ee530de366eac3cb5331c3a659400778adf95164b5ec16aff1"} Nov 01 22:07:10.631464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.631232 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podStartSLOduration=1.631211392 podStartE2EDuration="1.631211392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:10.574115014 +0000 UTC m=+1384.267095502" watchObservedRunningTime="2026-11-01 22:07:10.631211392 +0000 UTC m=+1384.324191878" Nov 01 22:07:10.966552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:10.966469 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:11.102038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.101998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:11.334412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.334378 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:07:11.342401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.342359 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.351764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.351735 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f2760d-openshift-default-dockercfg-fsbj7\"" Nov 01 22:07:11.356435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.356405 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:07:11.405054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.405017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:11.491467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.449200 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:11.491758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.486713 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.491975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.491955 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.505581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.505545 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.540472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.505815 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.540831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.540784 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.568468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.541149 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.594526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.574507 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2x28\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.594526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.574677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.594526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.574759 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796455 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2x28\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796644 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796864 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.796926 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.797031 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.797079 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.797154 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.798465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.797716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.799914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.799384 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.820470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.807708 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.820716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.810727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.820995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.810961 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.821135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.816524 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.859811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.840725 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:11.859811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.841030 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.859811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.848635 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.863962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.863932 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2x28\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:11.982945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:11.982902 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:12.214006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.213781 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:12.351787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.350611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:12.354591 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:12.354561 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4417b012_afea_41c9_8046_35f17e09053e.slice/crio-e81de78f090dadadcbd8ae2cabd00cf250cad3aa0cc007ee4818c843d70c0910 WatchSource:0}: Error finding container e81de78f090dadadcbd8ae2cabd00cf250cad3aa0cc007ee4818c843d70c0910: Status 404 returned error can't find the container with id e81de78f090dadadcbd8ae2cabd00cf250cad3aa0cc007ee4818c843d70c0910 Nov 01 22:07:12.354738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.354597 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:07:12.357766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.357733 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:12.357856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.357797 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:12.357856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.357828 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:12.797294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.797255 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerStarted","Data":"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c"} Nov 01 22:07:12.797529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.797314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerStarted","Data":"e81de78f090dadadcbd8ae2cabd00cf250cad3aa0cc007ee4818c843d70c0910"} Nov 01 22:07:12.797701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.797679 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:12.799505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.799435 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:07:12.856533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.856486 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podStartSLOduration=1.856470426 podStartE2EDuration="1.856470426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:12.825195771 +0000 UTC m=+1386.518176259" watchObservedRunningTime="2026-11-01 22:07:12.856470426 +0000 UTC m=+1386.549450912" Nov 01 22:07:12.984061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.984027 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:12.988957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:12.988925 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:13.147543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:13.143975 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:13.532238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:13.532198 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:13.991216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:13.991037 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:14.160232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:14.160194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:14.314761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:14.307805 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:14.819215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:14.819185 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:14.820059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:14.820037 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:07:14.923289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:14.921952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:15.432915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:15.432882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:15.562615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:15.562586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:16.048868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:16.048836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:16.625635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:16.625600 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:16.757739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:16.757702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:18.640720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:18.640635 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:18.780399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:18.780364 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:18.859643 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:07:18.859601 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:07:19.414507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:19.414194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:20.144716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:20.144681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:20.696222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:20.696185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:20.857075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:20.857041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:20.953422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:20.953325 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn"] Nov 01 22:07:20.999396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:20.999359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:21.003435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:21.003407 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb882c-xz7dn"] Nov 01 22:07:21.132139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:21.132098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:21.395183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:21.395152 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:22.288938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:22.288898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:22.406435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:22.406400 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:22.447526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:22.447490 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a49f3ff-9d30-44c7-bb66-6f7696fa8d54" path="/var/lib/kubelet/pods/0a49f3ff-9d30-44c7-bb66-6f7696fa8d54/volumes" Nov 01 22:07:23.438974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:23.438938 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:23.983940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:23.983909 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:07:24.564728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:24.564688 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:25.342024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:25.341980 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:25.776473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:25.776378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:26.488361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:26.488216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:26.676959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:26.676862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:26.825922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:26.825881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:28.370468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:28.368561 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-p4lpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:07:28.370468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:28.368641 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.475200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:29.475159 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:29.626467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:29.624982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:30.086471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.077580 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:30.471335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.471242 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.543501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.543458 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:07:30.544299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.544260 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" containerID="cri-o://90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc" gracePeriod=30 Nov 01 22:07:30.631300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.631261 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz"] Nov 01 22:07:30.639811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.639782 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.648547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.647134 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz"] Nov 01 22:07:30.648547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.648300 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:07:30.648547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.648404 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:07:30.670488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.665719 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:07:30.670488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.666044 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:07:30.684402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.684363 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47lz\" (UniqueName: \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.684681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.684661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.685403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.685362 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.787207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.787167 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.787473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.787436 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47lz\" (UniqueName: \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.787594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.787533 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.788266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.788243 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.789860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.789839 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.799566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.799546 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47lz\" (UniqueName: \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\") pod \"maas-api-revoke-keys-e2e-api-b-18657f-chsnz\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:30.969372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:30.969335 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:31.130058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.129992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:31.278465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.270617 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:31.294762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.294720 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz"] Nov 01 22:07:31.303502 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:31.302775 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff81e0a_e474_4ae2_b0d6_404b7ce5ce10.slice/crio-05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f WatchSource:0}: Error finding container 05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f: Status 404 returned error can't find the container with id 05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f Nov 01 22:07:31.409493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.409421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:31.566180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.566104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" event={"ID":"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10","Type":"ContainerStarted","Data":"05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f"} Nov 01 22:07:31.851517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.851403 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:07:31.857338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.857309 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.905199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.905101 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:07:31.913600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913568 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913706 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.913926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913906 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.914102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.913977 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:31.914102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:31.914071 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmng2\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.015812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.015777 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.015812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.015822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.015897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.015950 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016034 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016138 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016207 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016308 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmng2\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016381 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016719 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.016856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.016813 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.017164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.017141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.018656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.018629 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.019146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.019122 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.028207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.028185 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmng2\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.028410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.028388 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.210117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.207882 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:32.511656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.511625 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:07:32.511780 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:32.511723 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba92ad64_e063_4f45_acb7_7258c8dfc1e0.slice/crio-cdcd53c8c0c4d04b4eb1c39fb861e8a7621538daf57ba8beb95a6c8d92c233a2 WatchSource:0}: Error finding container cdcd53c8c0c4d04b4eb1c39fb861e8a7621538daf57ba8beb95a6c8d92c233a2: Status 404 returned error can't find the container with id cdcd53c8c0c4d04b4eb1c39fb861e8a7621538daf57ba8beb95a6c8d92c233a2 Nov 01 22:07:32.514894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.514861 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:32.515016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.514940 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:32.515016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.514972 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:32.535842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.535811 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:32.594193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.594124 5036 generic.go:356] "Generic (PLEG): container finished" podID="2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" containerID="b7cdf54ccd27310c853bfeacbdea8a9ce94ab6347b2fbda5cdd08bc9d8775c72" exitCode=0 Nov 01 22:07:32.594719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.594366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" event={"ID":"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10","Type":"ContainerDied","Data":"b7cdf54ccd27310c853bfeacbdea8a9ce94ab6347b2fbda5cdd08bc9d8775c72"} Nov 01 22:07:32.607010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.606977 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" event={"ID":"ba92ad64-e063-4f45-acb7-7258c8dfc1e0","Type":"ContainerStarted","Data":"cdcd53c8c0c4d04b4eb1c39fb861e8a7621538daf57ba8beb95a6c8d92c233a2"} Nov 01 22:07:32.828020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.827938 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:32.971985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:32.971936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:33.619042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.619001 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" event={"ID":"ba92ad64-e063-4f45-acb7-7258c8dfc1e0","Type":"ContainerStarted","Data":"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677"} Nov 01 22:07:33.644269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.644216 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podStartSLOduration=2.644202008 podStartE2EDuration="2.644202008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:33.642153229 +0000 UTC m=+1407.335133730" watchObservedRunningTime="2026-11-01 22:07:33.644202008 +0000 UTC m=+1407.337182493" Nov 01 22:07:33.849875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.849848 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.885885 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.888564 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\") pod \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.888733 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\" (UniqueName: \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\") pod \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.888898 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\") pod \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\" (UID: \"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10\") " Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.890540 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca" pod "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" (UID: "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:07:33.912071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.892093 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca" pod "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" (UID: "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:07:33.924799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.924758 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz" pod "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" (UID: "2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10"). InnerVolumeSpecName "kube-api-access-h47lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:33.932130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.930502 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:33.932130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.930542 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:33.932130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:33.930558 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47lz\" (UniqueName: \"kubernetes.io/projected/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10-kube-api-access-h47lz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:34.211077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.210575 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:34.211077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.210695 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:34.211077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.210742 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:34.644663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.641469 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" Nov 01 22:07:34.644663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.641632 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz" event={"ID":"2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10","Type":"ContainerDied","Data":"05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f"} Nov 01 22:07:34.644663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:34.641674 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ed8aae8f93769e3f71d386039d00042c8279bd1f398667232fb5261206ce7f" Nov 01 22:07:35.013353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:35.013257 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:35.209030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:35.208988 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:35.209224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:35.209057 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:35.720140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:35.720107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:36.039930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.038873 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:07:36.209151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.209055 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:36.209151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.209135 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:36.222502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222467 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222566 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222813 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222887 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.222952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.222889 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:07:36.223064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223012 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:07:36.223064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223040 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.223163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223109 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\") pod \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\" (UID: \"5a4c32b4-f484-4287-ba7b-ba56a80d1e55\") " Nov 01 22:07:36.223163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223121 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:07:36.223260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223139 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:07:36.223319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223259 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:07:36.223792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223761 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.223893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223796 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.223893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223814 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.223893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223827 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.223893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.223841 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.225618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.225588 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:07:36.225738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.225720 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:36.225850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.225831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "kube-api-access-6mzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:36.226369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.226351 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo" pod "5a4c32b4-f484-4287-ba7b-ba56a80d1e55" (UID: "5a4c32b4-f484-4287-ba7b-ba56a80d1e55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:07:36.325952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.325906 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mzs5\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-kube-api-access-6mzs5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.325952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.325942 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.325952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.325955 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.326214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.325969 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4c32b4-f484-4287-ba7b-ba56a80d1e55-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:36.458742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.458707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:36.662113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.662083 5036 generic.go:356] "Generic (PLEG): container finished" podID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerID="90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc" exitCode=0 Nov 01 22:07:36.662306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.662136 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" Nov 01 22:07:36.662306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.662165 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerDied","Data":"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc"} Nov 01 22:07:36.662306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.662218 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz" event={"ID":"5a4c32b4-f484-4287-ba7b-ba56a80d1e55","Type":"ContainerDied","Data":"5b127435359df47896aa4695f88b2fa08099c17e8c3d4c0c34128ef817b655b5"} Nov 01 22:07:36.662306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.662242 5036 scope.go:122] "RemoveContainer" containerID="90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc" Nov 01 22:07:36.674061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.673876 5036 scope.go:122] "RemoveContainer" containerID="10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123" Nov 01 22:07:36.720015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.719919 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:07:36.727276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.727249 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bb882c-openshift-default-64f5fbfd9f-8lkjz"] Nov 01 22:07:36.729368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.729347 5036 scope.go:122] "RemoveContainer" containerID="90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc" Nov 01 22:07:36.729794 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:07:36.729770 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc\": container with ID starting with 90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc not found: ID does not exist" containerID="90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc" Nov 01 22:07:36.729844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.729810 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc"} err="failed to get container status \"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc\": rpc error: code = NotFound desc = could not find container \"90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc\": container with ID starting with 90a8e5a58da26a202d751720b5670951c45753af8503d48db1196046300bbdbc not found: ID does not exist" Nov 01 22:07:36.729844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.729837 5036 scope.go:122] "RemoveContainer" containerID="10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123" Nov 01 22:07:36.730125 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:07:36.730103 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123\": container with ID starting with 10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123 not found: ID does not exist" containerID="10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123" Nov 01 22:07:36.730169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:36.730134 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123"} err="failed to get container status \"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123\": rpc error: code = NotFound desc = could not find container \"10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123\": container with ID starting with 10204aa5d49cfbaf09276cb4a18e2b8267c591b0f7d846721e8b51d7362cb123 not found: ID does not exist" Nov 01 22:07:37.237492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:37.234207 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:37.237492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:37.234293 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:37.253341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:37.253280 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:37.399175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:37.395471 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:37.528974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:37.528854 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:38.212574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:38.212536 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:38.213161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:38.213130 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:38.431668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:38.431557 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:38.446605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:38.446544 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" path="/var/lib/kubelet/pods/5a4c32b4-f484-4287-ba7b-ba56a80d1e55/volumes" Nov 01 22:07:39.211922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:39.211881 5036 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:07:39.212108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:39.211964 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 01 22:07:39.228949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:39.228917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:40.132035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.131996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:40.234261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.234229 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:40.235239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.235208 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:40.235896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.235875 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:07:40.273478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.271188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:40.994556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:40.994523 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:41.123354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.123313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:41.247588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.247499 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:07:41.274468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.249270 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.274722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.274692 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.275201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.275178 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" containerName="revoke-keys" Nov 01 22:07:41.275336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.275320 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" containerName="revoke-keys" Nov 01 22:07:41.275460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.275430 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.275573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.275559 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.276205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.276184 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.316760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.316726 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" containerName="revoke-keys" Nov 01 22:07:41.336393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.336360 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4c32b4-f484-4287-ba7b-ba56a80d1e55" containerName="istio-proxy" Nov 01 22:07:41.361502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.355971 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.368483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.368425 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17957f-openshift-default-dockercfg-5jkrs\"" Nov 01 22:07:41.374763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.374730 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:07:41.415660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.392962 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:07:41.444498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.442705 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.448775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.447297 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:07:41.583761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.583717 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.583773 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.583853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.583888 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.583926 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584000 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584065 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c47\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584503 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584564 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfvw\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584664 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584695 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.584906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.584906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.584812 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.616469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.598008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701492 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c47\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701566 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701726 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfvw\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701749 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701803 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701831 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701862 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701917 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701947 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.701981 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702112 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702202 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702239 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702281 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.707482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.702366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.715488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.713180 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.715488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.714231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.728481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.725196 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.728481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.726645 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.728481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.727234 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.728481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.728004 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.728890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.728530 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.728890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.728771 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.734973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.734944 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.735292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.735267 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.763594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.763510 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfvw\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.764276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.764215 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.765846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.765785 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.766411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.766359 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.851556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.835101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:41.851556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.835803 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.851556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.841273 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c47\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:41.864897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:41.864845 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:42.040756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.040723 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:42.070775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.070747 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:42.374314 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:42.374269 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941318d2_f54e_4934_9c9c_7267bcc75f54.slice/crio-ef1b0451d54df3276cc0e1f0ac685e5c89c3a7f52dd6982a1bf5469c2f86279b WatchSource:0}: Error finding container ef1b0451d54df3276cc0e1f0ac685e5c89c3a7f52dd6982a1bf5469c2f86279b: Status 404 returned error can't find the container with id ef1b0451d54df3276cc0e1f0ac685e5c89c3a7f52dd6982a1bf5469c2f86279b Nov 01 22:07:42.375438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.375413 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:07:42.403520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.395885 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.403520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.395982 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.403520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.396028 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.432671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.432494 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:42.506473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.476425 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:07:42.506473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.482746 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.506473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.482836 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.506473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.482879 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:42.866928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.866892 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" event={"ID":"50cd8a73-70f3-448a-9dc8-2598d4a2c52f","Type":"ContainerStarted","Data":"8592cd47fb99a061529b2c7b37cbc53e5bd93e566b3603d9e272f122c7dfe98a"} Nov 01 22:07:42.867138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.866950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" event={"ID":"50cd8a73-70f3-448a-9dc8-2598d4a2c52f","Type":"ContainerStarted","Data":"55c932b4cb32e771356dcdbaac26c93461e4175be891386c76bdbeb80b5ffe2a"} Nov 01 22:07:42.875474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.873725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" event={"ID":"941318d2-f54e-4934-9c9c-7267bcc75f54","Type":"ContainerStarted","Data":"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1"} Nov 01 22:07:42.875474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.873798 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" event={"ID":"941318d2-f54e-4934-9c9c-7267bcc75f54","Type":"ContainerStarted","Data":"ef1b0451d54df3276cc0e1f0ac685e5c89c3a7f52dd6982a1bf5469c2f86279b"} Nov 01 22:07:42.974163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:42.974091 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podStartSLOduration=1.9740739 podStartE2EDuration="1.9740739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:42.961815732 +0000 UTC m=+1416.654796222" watchObservedRunningTime="2026-11-01 22:07:42.9740739 +0000 UTC m=+1416.667054387" Nov 01 22:07:43.041099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.041058 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:43.041565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.041510 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podStartSLOduration=2.041494491 podStartE2EDuration="2.041494491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:42.986723135 +0000 UTC m=+1416.679703634" watchObservedRunningTime="2026-11-01 22:07:43.041494491 +0000 UTC m=+1416.734474977" Nov 01 22:07:43.042548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.042516 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:43.042664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.042572 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:43.071404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.071372 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:43.073538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.073507 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:43.073664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.073565 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:43.445607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.445572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:43.701525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.700991 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:43.835085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:43.835050 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:44.047312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.047275 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:44.047687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.047648 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:44.071922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.071890 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:44.072110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.071939 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:44.193228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.193173 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:45.025476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:44.991235 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:45.088465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.086105 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:45.088465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.086166 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:45.090757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.089942 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:45.090757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.090007 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:45.254496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.250007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.302398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.302297 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.311595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.311524 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.422381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.422329 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.423143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.423096 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:07:45.689632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.689518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.719626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.698515 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:45.841583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.841465 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:45.948540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:45.947783 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:46.067611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.065857 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:46.067611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.065915 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:46.112470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.090790 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:46.112470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.090848 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:46.250440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.250262 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:46.251295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.251275 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:07:46.526526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.511985 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:07:46.885537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.885340 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:46.892475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.887317 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:46.938556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.913995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:07:46.942476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.939955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:46.967870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.952555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:46.967870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:46.956522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:47.068480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.061064 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:07:47.068480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.061128 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:07:47.068480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.061171 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:07:47.068480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.063337 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:47.068480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.063385 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:47.072916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.072888 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:47.073146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.073121 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:47.419477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.386553 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:47.531681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.531643 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:07:47.566808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.565542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:47.652967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.652933 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:47.655917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.655895 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:07:47.801364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:47.799827 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:07:48.051676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.051570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:48.054209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.054179 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:07:48.056237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.056203 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:48.064801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.064074 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:48.064988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.064963 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:48.071758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.071735 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:48.107706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.107661 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:48.113467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.076063 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:07:48.539427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:48.539385 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:48.952524 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:07:48.951360 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:07:49.047470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:49.041993 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:49.047470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:49.042063 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:49.071322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:49.071288 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:49.071559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:49.071343 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:49.648511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:49.648473 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:50.041473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:50.041057 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 01 22:07:50.041473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:50.041129 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 01 22:07:50.071104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:50.071066 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 01 22:07:50.071332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:50.071305 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 01 22:07:50.514417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:50.514312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:51.048307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.048271 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:51.049375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.049350 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:51.050639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.050611 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:07:51.079976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.079939 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:51.081956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.081935 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:51.084196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.084084 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:07:51.255696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.255632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:51.951606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:51.951556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:52.086819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.086777 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:52.115586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.115557 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:07:52.115757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.115621 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:52.643653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.643615 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:52.776301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.776260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:52.957570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:52.957483 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:53.764630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:53.764588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:54.529948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.529915 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:54.688329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.688231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:54.740149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.740117 5036 generic.go:356] "Generic (PLEG): container finished" podID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerID="b2221105533cf001c0e12b39ea645ba2eea7939d0bae75acadb001377e057166" exitCode=0 Nov 01 22:07:54.740339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.740194 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerDied","Data":"b2221105533cf001c0e12b39ea645ba2eea7939d0bae75acadb001377e057166"} Nov 01 22:07:54.740792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.740773 5036 scope.go:122] "RemoveContainer" containerID="b2221105533cf001c0e12b39ea645ba2eea7939d0bae75acadb001377e057166" Nov 01 22:07:54.742954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.742922 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:54.743054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.742993 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:54.743054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:54.743026 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:07:55.533725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.533685 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:55.752579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.752545 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerStarted","Data":"ced928bbe7bc22e4f0a95eb58b4027471e076da816652342e1c3d4c09302d26f"} Nov 01 22:07:55.926946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.926819 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:07:55.954340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.954312 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:07:55.954623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.954598 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:55.961676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.961649 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d0a5de\"/\"openshift-service-ca.crt\"" Nov 01 22:07:55.962250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.962230 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d0a5de\"/\"test-model-d0a5de-kserve-self-signed-certs\"" Nov 01 22:07:55.964607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.962401 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d0a5de\"/\"kube-root-ca.crt\"" Nov 01 22:07:55.965142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.965109 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d0a5de\"/\"default-dockercfg-5k4nz\"" Nov 01 22:07:55.995115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:55.995084 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:07:56.040715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.003175 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:07:56.040996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.040968 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:07:56.041331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.036198 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:56.156993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.156949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.157362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.157339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.157575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.157556 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.157722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.157704 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjbn\" (UniqueName: \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.157935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.157917 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.163302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.163273 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.270656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.270719 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjbn\" (UniqueName: \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.270757 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.270960 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.271060 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.271222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.271798 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.272845 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.280520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.273462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.304502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.299882 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjbn\" (UniqueName: \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.304502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.300678 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.311466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.307059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\") pod \"test-model-d0a5de-kserve-69fb759c49-n874m\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:56.540707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.540474 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:56.575917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:56.575313 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:57.003521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.001040 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:07:57.003521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.001094 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:07:57.027552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.018470 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:07:57.030943 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:07:57.030898 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151bc2b8_bc39_478c_8a20_ea4e1c20e7ad.slice/crio-2c190de2ecdbc5fe78e6e7a1a147adb770fd29172b34c4ac8d8ff6d8cdc57935 WatchSource:0}: Error finding container 2c190de2ecdbc5fe78e6e7a1a147adb770fd29172b34c4ac8d8ff6d8cdc57935: Status 404 returned error can't find the container with id 2c190de2ecdbc5fe78e6e7a1a147adb770fd29172b34c4ac8d8ff6d8cdc57935 Nov 01 22:07:57.259309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.259162 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:57.857474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.854369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:57.860136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.860102 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" event={"ID":"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad","Type":"ContainerStarted","Data":"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf"} Nov 01 22:07:57.860339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.860324 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" event={"ID":"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad","Type":"ContainerStarted","Data":"2c190de2ecdbc5fe78e6e7a1a147adb770fd29172b34c4ac8d8ff6d8cdc57935"} Nov 01 22:07:57.975609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:57.975541 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podStartSLOduration=2.975518654 podStartE2EDuration="2.975518654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:57.886786705 +0000 UTC m=+1431.579767192" watchObservedRunningTime="2026-11-01 22:07:57.975518654 +0000 UTC m=+1431.668499142" Nov 01 22:07:58.016102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.008580 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:07:58.016629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.016588 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:07:58.233551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.227964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:58.393021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.392851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:58.576250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.576218 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:07:58.607516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.604545 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:58.996251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.995767 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:07:58.996251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:58.995822 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:07:59.213297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.213257 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:59.344927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.344886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:59.553747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.553715 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:07:59.563338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.563312 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.640053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.639967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:59.640951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.640927 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e2a91d-openshift-default-dockercfg-htpfz\"" Nov 01 22:07:59.643010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.642988 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:07:59.648928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.648897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.648991 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649176 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmqc\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649286 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649604 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.649722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.649651 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779409 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779468 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779519 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779570 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmqc\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779660 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779731 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779786 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.779825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.785488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.780352 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.858504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.824842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.919866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.877812 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.919866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.879842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.919866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.879872 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.919866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.883570 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:59.919866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.896797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.930346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.930273 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.932757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.932703 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:07:59.997270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:07:59.997232 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmqc\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\") pod \"e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:00.062681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.016945 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:08:00.062681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.017010 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:08:00.262506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.262425 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:00.335277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.335046 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:00.720849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.719987 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:08:00.725749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.724304 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:00.725749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.724398 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:00.725749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:00.724459 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:01.009782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.009745 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:08:01.010374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.010341 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:08:01.067315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.067216 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerStarted","Data":"61211256e61c64af7bee645462b01d381845ef03a2bc55062bef8d6d065f6a25"} Nov 01 22:08:01.067610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.067575 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerStarted","Data":"7eef58cbbcecca0fd2bb072dd27cac334660c3cfee6958830416b0cfe6409d85"} Nov 01 22:08:01.125478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.120960 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podStartSLOduration=2.120938454 podStartE2EDuration="2.120938454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:01.097748372 +0000 UTC m=+1434.790728871" watchObservedRunningTime="2026-11-01 22:08:01.120938454 +0000 UTC m=+1434.813918941" Nov 01 22:08:01.275523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.264894 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:01.275523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.265037 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:08:01.275523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.265080 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:08:01.290521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.287841 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:01.995430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.995395 5036 patch_prober.go:29] interesting pod/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:08:01.995688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:01.995475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 22:08:02.032888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:02.032853 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm"] Nov 01 22:08:02.041395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:02.041364 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d09db57e-8kbpm"] Nov 01 22:08:02.268431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:02.268358 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:02.405765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:02.405644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:02.447237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:02.447188 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866" path="/var/lib/kubelet/pods/e0e1c4ac-59a2-4cf4-80a0-d0b58d9a8866/volumes" Nov 01 22:08:03.000299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.000269 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:08:03.000708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.000683 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:08:03.001358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.001342 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:08:03.096733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.096698 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:03.097917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.097770 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:08:03.222672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.222636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:03.440793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.440756 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:03.573323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:03.573287 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:04.036008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:04.035969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:04.668286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:04.668249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:04.802046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:04.802007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:04.955595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:04.950086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:05.973464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:05.968483 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:06.087215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.087178 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:06.480838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.480803 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:06.575838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.575805 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:06.593830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.593797 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:06.732869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.732768 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:06.827847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.827815 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:08:06.835246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.835170 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:06.847546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.847517 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:08:06.848639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.848616 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8483c2-openshift-default-dockercfg-nw2v7\"" Nov 01 22:08:06.902465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:06.896064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:07.094468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094373 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.094468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094436 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094615 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094774 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxsf\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094803 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094859 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.095087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.094983 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.162552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.107575 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242490 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242552 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242616 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242842 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxsf\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242869 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.242925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.243032 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.243601 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.244189 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.256534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.255814 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.296672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.274919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.296672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.275643 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.296672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.288125 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.316466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.315150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxsf\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.370785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.356939 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.379303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.379235 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.485620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.484176 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:07.491303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.490903 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:07.741653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.741534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:07.851689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.848880 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:08:07.854621 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:07.854512 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40b676b_5ad7_423d_9223_abed4d646fee.slice/crio-30479df2115124d2aa9e7eb68913c0a3494e6319e15d7e67190253a32d95d83d WatchSource:0}: Error finding container 30479df2115124d2aa9e7eb68913c0a3494e6319e15d7e67190253a32d95d83d: Status 404 returned error can't find the container with id 30479df2115124d2aa9e7eb68913c0a3494e6319e15d7e67190253a32d95d83d Nov 01 22:08:07.860402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.860364 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:07.860501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.860476 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:07.860553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:07.860525 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:08.394098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.394056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerStarted","Data":"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368"} Nov 01 22:08:08.394746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.394713 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerStarted","Data":"30479df2115124d2aa9e7eb68913c0a3494e6319e15d7e67190253a32d95d83d"} Nov 01 22:08:08.426024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.424953 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podStartSLOduration=2.424935533 podStartE2EDuration="2.424935533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:08.419300712 +0000 UTC m=+1442.112281221" watchObservedRunningTime="2026-11-01 22:08:08.424935533 +0000 UTC m=+1442.117916010" Nov 01 22:08:08.492455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.492407 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:08.498495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.498467 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:08.601421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:08.601386 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:09.312219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:09.312180 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:09.412160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:09.411974 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:09.416970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:09.416940 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:08:10.132004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:10.131972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:10.652455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:10.652394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:10.855736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:10.855687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:11.003345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:11.003248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:11.890492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:11.890392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:12.076477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:12.055265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:12.943252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:12.943214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:13.734480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:13.729497 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:08:13.798472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:13.793871 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:14.053478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.050677 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:14.053478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.052033 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:08:14.053478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.052145 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.105511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.101079 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9d8b5d\"/\"kube-root-ca.crt\"" Nov 01 22:08:14.105511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.101079 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9d8b5d\"/\"openshift-service-ca.crt\"" Nov 01 22:08:14.105511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.101154 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9d8b5d\"/\"test-model-9d8b5d-kserve-self-signed-certs\"" Nov 01 22:08:14.105511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.101229 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9d8b5d\"/\"default-dockercfg-nfthh\"" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.121509 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvqq\" (UniqueName: \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.121595 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.121633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.121673 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.121818 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.123498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.122152 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226376 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvqq\" (UniqueName: \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226599 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.226659 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.228467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.227269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.229000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.228479 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.229000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.228732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.247669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.239228 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.247669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.241748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.263259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.263200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvqq\" (UniqueName: \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\") pod \"test-model-9d8b5d-kserve-59cbd98bb7-x4pkd\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.424232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.424140 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:14.722875 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:14.722842 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3a2eb43_2e70_4f89_8aea_ad68d029d500.slice/crio-588429239aa18447369d51ee49f72accb06c444fcef747117527f3701c94bf7a WatchSource:0}: Error finding container 588429239aa18447369d51ee49f72accb06c444fcef747117527f3701c94bf7a: Status 404 returned error can't find the container with id 588429239aa18447369d51ee49f72accb06c444fcef747117527f3701c94bf7a Nov 01 22:08:14.724811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:14.724774 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:08:15.157947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:15.157917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:15.509672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:15.509576 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" event={"ID":"a3a2eb43-2e70-4f89-8aea-ad68d029d500","Type":"ContainerStarted","Data":"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093"} Nov 01 22:08:15.509672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:15.509626 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" event={"ID":"a3a2eb43-2e70-4f89-8aea-ad68d029d500","Type":"ContainerStarted","Data":"588429239aa18447369d51ee49f72accb06c444fcef747117527f3701c94bf7a"} Nov 01 22:08:15.533813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:15.533762 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podStartSLOduration=2.533747603 podStartE2EDuration="2.533747603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:15.528910495 +0000 UTC m=+1449.221890983" watchObservedRunningTime="2026-11-01 22:08:15.533747603 +0000 UTC m=+1449.226728088" Nov 01 22:08:16.007467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.007109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:16.234468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.229264 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:16.353467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.352955 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:16.434469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.431527 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:16.574505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.572791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:16.701730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.700493 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz"] Nov 01 22:08:16.733467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.730598 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18657f-chsnz"] Nov 01 22:08:16.914508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:16.914472 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:17.162820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:17.162783 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:18.114178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.114141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:18.440334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.440259 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10" path="/var/lib/kubelet/pods/2ff81e0a-e474-4ae2-b0d6-404b7ce5ce10/volumes" Nov 01 22:08:18.663387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.663353 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:08:18.681169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.681141 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:08:18.681407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.681237 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.686530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.686507 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17957f\"/\"default-dockercfg-97nmd\"" Nov 01 22:08:18.689362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.689342 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-17957f\"/\"test-model-17957f-kserve-self-signed-certs\"" Nov 01 22:08:18.721847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.719332 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17957f\"/\"openshift-service-ca.crt\"" Nov 01 22:08:18.721847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.719458 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-17957f\"/\"kube-root-ca.crt\"" Nov 01 22:08:18.744094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.744066 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:18.790462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.790406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.790633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.790607 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxqm\" (UniqueName: \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.790682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.790666 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.790941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.790918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.790989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.790953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.791106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.791090 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.895611 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.895871 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.895975 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxqm\" (UniqueName: \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.896018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.896241 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.898470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.896265 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.921464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.913840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.931765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.931731 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.931988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.931941 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.936249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.936202 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.960881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.960784 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxqm\" (UniqueName: \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:18.976482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:18.976034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\") pod \"test-model-17957f-kserve-855ddd64d5-f72qz\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:19.031360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.030619 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:19.083936 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:08:19.083865 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:08:19.446892 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:19.446858 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34e4fc1d_d15d_42d6_a556_3ec303761bd2.slice/crio-6c2af65647304432fa0c877aadb8f4e14936c72ade3354c76c6a83f0ba032ccf WatchSource:0}: Error finding container 6c2af65647304432fa0c877aadb8f4e14936c72ade3354c76c6a83f0ba032ccf: Status 404 returned error can't find the container with id 6c2af65647304432fa0c877aadb8f4e14936c72ade3354c76c6a83f0ba032ccf Nov 01 22:08:19.446892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.446874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:08:19.513568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.513373 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:19.651615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.651581 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" event={"ID":"34e4fc1d-d15d-42d6-a556-3ec303761bd2","Type":"ContainerStarted","Data":"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3"} Nov 01 22:08:19.651820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.651627 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" event={"ID":"34e4fc1d-d15d-42d6-a556-3ec303761bd2","Type":"ContainerStarted","Data":"6c2af65647304432fa0c877aadb8f4e14936c72ade3354c76c6a83f0ba032ccf"} Nov 01 22:08:19.677895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:19.677837 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podStartSLOduration=1.677818298 podStartE2EDuration="1.677818298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:19.673325398 +0000 UTC m=+1453.366305886" watchObservedRunningTime="2026-11-01 22:08:19.677818298 +0000 UTC m=+1453.370798788" Nov 01 22:08:20.031290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:20.031255 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:20.630091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:20.630053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:21.258163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.258126 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:08:21.270575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.270517 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:08:21.270776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.270644 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.279455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.278188 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-830149\"/\"default-dockercfg-gw229\"" Nov 01 22:08:21.279455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.278299 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-830149\"/\"test-model-830149-kserve-self-signed-certs\"" Nov 01 22:08:21.294140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.292932 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-830149\"/\"openshift-service-ca.crt\"" Nov 01 22:08:21.294140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.293041 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-830149\"/\"kube-root-ca.crt\"" Nov 01 22:08:21.336548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.336508 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:21.384553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384515 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.384744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384634 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4kr\" (UniqueName: \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.384744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384709 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.384744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384739 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.384873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.384937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.384920 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.476476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.470627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.486684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.486874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4kr\" (UniqueName: \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.486957 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.486983 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.487012 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.487333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.487076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.488231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.488157 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.494778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.489020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.494778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.489277 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.553468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.552817 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4kr\" (UniqueName: \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.576470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.559947 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.576470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.564027 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\") pod \"test-model-830149-kserve-f44787b85-cc6wd\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:21.604486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:21.603744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:22.075865 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:22.075831 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3d355a_afae_4e36_84db_c10434e04638.slice/crio-99c30e06a3bbe1b26368289d5603547403d23c8efbe0e9992e908e0b71eec197 WatchSource:0}: Error finding container 99c30e06a3bbe1b26368289d5603547403d23c8efbe0e9992e908e0b71eec197: Status 404 returned error can't find the container with id 99c30e06a3bbe1b26368289d5603547403d23c8efbe0e9992e908e0b71eec197 Nov 01 22:08:22.076985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.076878 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:08:22.200400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.200363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:22.810988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.810937 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" event={"ID":"0b3d355a-afae-4e36-84db-c10434e04638","Type":"ContainerStarted","Data":"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b"} Nov 01 22:08:22.811238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.811004 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" event={"ID":"0b3d355a-afae-4e36-84db-c10434e04638","Type":"ContainerStarted","Data":"99c30e06a3bbe1b26368289d5603547403d23c8efbe0e9992e908e0b71eec197"} Nov 01 22:08:22.834910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.834851 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podStartSLOduration=1.8348346420000001 podStartE2EDuration="1.834834642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:22.832713434 +0000 UTC m=+1456.525693921" watchObservedRunningTime="2026-11-01 22:08:22.834834642 +0000 UTC m=+1456.527815129" Nov 01 22:08:22.861213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:22.861183 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:23.534768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.534732 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:23.607041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.607003 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:23.663457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.663408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:23.808032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.805711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:23.808032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.807826 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:08:23.808264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.808069 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" containerID="cri-o://ced928bbe7bc22e4f0a95eb58b4027471e076da816652342e1c3d4c09302d26f" gracePeriod=30 Nov 01 22:08:23.962248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:23.962214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:24.427342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:24.427306 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:24.464947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:24.464895 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:24.537848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:24.537811 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:08:24.622009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:24.621946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:25.250037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.250000 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:25.323537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.320503 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:08:25.343676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.343646 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.349659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.349604 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:08:25.355438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.355412 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-487639-openshift-default-dockercfg-hcp7z\"" Nov 01 22:08:25.360547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.360471 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.485351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.485620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.485811 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.485865 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.485996 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.486203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.486277 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.486384 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.486538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.486513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psths\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.529525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.512184 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.589751 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.589870 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590005 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590113 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590250 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590286 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590351 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.590428 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psths\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.591745 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.592799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.593412 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.594519 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.596469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.594759 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.631488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.627846 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psths\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.645463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.639970 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.645463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.643013 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.660858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.657108 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:25.671467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.670328 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:25.696469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:25.689777 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:26.286662 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:26.286629 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8a94d1_7827_40e9_ae5f_139ac8b63092.slice/crio-430f7dbd692b4377e849ee7e9176659c3b7dde34a67237eeb079164226126180 WatchSource:0}: Error finding container 430f7dbd692b4377e849ee7e9176659c3b7dde34a67237eeb079164226126180: Status 404 returned error can't find the container with id 430f7dbd692b4377e849ee7e9176659c3b7dde34a67237eeb079164226126180 Nov 01 22:08:26.287711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:26.287666 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:08:26.290126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:26.290092 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:26.290250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:26.290154 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:26.290250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:26.290188 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:26.323072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:26.323043 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:27.279536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.268971 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerStarted","Data":"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c"} Nov 01 22:08:27.279536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.269057 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerStarted","Data":"430f7dbd692b4377e849ee7e9176659c3b7dde34a67237eeb079164226126180"} Nov 01 22:08:27.391473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.387914 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podStartSLOduration=2.387892091 podStartE2EDuration="2.387892091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:27.332507025 +0000 UTC m=+1461.025487512" watchObservedRunningTime="2026-11-01 22:08:27.387892091 +0000 UTC m=+1461.080872578" Nov 01 22:08:27.440468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.424550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:27.556634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.529529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.556634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.550544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.588464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.585151 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": context deadline exceeded" Nov 01 22:08:27.632671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.632621 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.695215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.695182 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:27.695543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.695522 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:08:27.695735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.695710 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:08:27.711466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.707589 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.814521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.813668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.894470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.890731 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.918466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.906879 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:27.959469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:27.948526 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:28.087524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.079859 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.108010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.107894 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.384527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.373668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.423465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.418798 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.526484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.505592 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.549470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.544531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.573692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.573659 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:08:28.575628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.575593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:08:28.576267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.576248 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:08:28.606163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.606129 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:28.667289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.667188 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:28.668095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.668070 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:08:28.692570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.692536 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:08:28.692830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.692803 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:08:28.717468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.708154 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.823253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.817527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:28.848565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.823914 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.084928 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.084968 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.891746 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.892336 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.915350 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:28.966790 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.089145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.089197 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.089235 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.015901 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.091126 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.103802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.091346 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:08:29.143049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.120572 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:29.143049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.122370 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:08:29.540514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.534654 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:29.540514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.535112 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:29.574465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.572038 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:08:29.574465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.572283 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:29.574465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.572545 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:08:29.579559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.579518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:08:29.675581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.675534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:29.700833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.694988 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.700833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.695593 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:29.799279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.799249 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:29.829178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.829135 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:29.845958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.845916 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.011848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:29.999735 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:30.038622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.031683 5036 generic.go:356] "Generic (PLEG): container finished" podID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerID="ced928bbe7bc22e4f0a95eb58b4027471e076da816652342e1c3d4c09302d26f" exitCode=0 Nov 01 22:08:30.038622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.031794 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerDied","Data":"ced928bbe7bc22e4f0a95eb58b4027471e076da816652342e1c3d4c09302d26f"} Nov 01 22:08:30.038622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.031880 5036 scope.go:122] "RemoveContainer" containerID="b2221105533cf001c0e12b39ea645ba2eea7939d0bae75acadb001377e057166" Nov 01 22:08:30.078569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.046407 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:30.078569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.050991 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:08:30.078569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.075662 5036 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.102678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.102633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.104578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.104510 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.105545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.105520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.106542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.106517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.107314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.107291 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:30.108174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.108152 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:30.153524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.144570 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:08:30.153524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.144668 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:08:30.153524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.147180 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.569819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.555698 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.597124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.583947 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.597124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.584384 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.597124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.584686 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:08:30.708159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.708104 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 01 22:08:30.946764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.754158 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:30.946764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.754559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:30.946764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.754617 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.946764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.843346 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:30.950908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:30.885605 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.154744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.129423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148458 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148567 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148608 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.212995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.148629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.398007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.385673 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:08:31.447499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.439897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:31.447499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.440011 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mrv7l" podUID="e20c269e-61ab-4af3-b1bf-eba3a120df3a" containerName="proxy" probeResult="failure" output="Get \"https://10.132.0.46:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.523664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.523628 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.571058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.571012 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:31.893943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.893837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:31.920678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:31.898501 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:32.008212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.008120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:32.105562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.105507 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:32.107253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.107215 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:32.273309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.273264 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:32.302143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.302103 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:32.303205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.303172 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:32.328260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.328223 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:32.608120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.608003 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:32.631773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.612352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:08:32.678410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.671808 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:32.703192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.677940 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:32.703192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.678148 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:32.703192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.678201 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:32.703192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.678708 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:32.778052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.778008 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 22:08:32.778383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.778359 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:32.909032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.908932 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:32.916664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.916633 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:08:32.917243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.917210 5036 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:32.978796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:32.978742 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:33.227560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.168114 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.527350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.191207 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:33.528078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.295546 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.528434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.328761 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:33.589111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.328820 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.589887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.328843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:33.590132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.485338 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.590395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.590358 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.590601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.590569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.590866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.590835 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.591058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.591028 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.591241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.591214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.591418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.591385 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.602782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.486316 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:08:33.602928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.602749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:08:33.635688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.612950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:08:33.675473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.666470 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:08:33.712702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.705887 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:08:33.718692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.718653 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:08:33.720500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.720249 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "kube-api-access-7qpnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:08:33.753473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.741691 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:08:33.753473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.743673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.753473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.743825 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\") pod \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\" (UID: \"b9da3113-b386-46e0-9ad1-cd0f3915d26d\") " Nov 01 22:08:33.753473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.744684 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:08:33.753473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.744923 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket" pod "b9da3113-b386-46e0-9ad1-cd0f3915d26d" (UID: "b9da3113-b386-46e0-9ad1-cd0f3915d26d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:08:33.783684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.783552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.847643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.783948 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:33.916517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.783989 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.916893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.800095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:33.917020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.802175 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917013 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917030 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917048 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917064 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917079 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da3113-b386-46e0-9ad1-cd0f3915d26d-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917093 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qpnz\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-kube-api-access-7qpnz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.917109 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:33.917136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.836625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:33.917518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.836523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:33.917518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.836563 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 01 22:08:33.917518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.836592 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:33.983492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.918472 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.001704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:33.933502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.026469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.024497 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da3113-b386-46e0-9ad1-cd0f3915d26d-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:34.034862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.034698 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" Nov 01 22:08:34.064013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.063752 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz" event={"ID":"b9da3113-b386-46e0-9ad1-cd0f3915d26d","Type":"ContainerDied","Data":"840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf"} Nov 01 22:08:34.064013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.063829 5036 scope.go:122] "RemoveContainer" containerID="ced928bbe7bc22e4f0a95eb58b4027471e076da816652342e1c3d4c09302d26f" Nov 01 22:08:34.197348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.197006 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.280651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.280584 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:08:34.338478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.337583 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.356465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.352527 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18657f-openshift-default-54fcbd9989-5w5bz"] Nov 01 22:08:34.424134 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:08:34.424090 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da3113_b386_46e0_9ad1_cd0f3915d26d.slice/crio-840a763029fac4258ffd515afd8a0a640163ad8c7866bd16e256a00e2d47fbbf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da3113_b386_46e0_9ad1_cd0f3915d26d.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:08:34.525648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.507822 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" path="/var/lib/kubelet/pods/b9da3113-b386-46e0-9ad1-cd0f3915d26d/volumes" Nov 01 22:08:34.530044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.527296 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.530044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.527773 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.532421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.532324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:34.616531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.615190 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:34.634709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.634633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.649967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.641817 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:34.670093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.669534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.697187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.696519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.805584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.805532 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.854468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.853536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.918692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.918593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.919374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.918920 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.943959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.918954 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.945189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.918978 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:34.968164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.919009 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:34.968669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.968644 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:35.004373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:34.984774 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.070368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.067142 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:35.375188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.359610 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.466476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.460548 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": context deadline exceeded" Nov 01 22:08:35.466476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.460538 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.475397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.472165 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:35.592052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.592007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:08:35.617465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.592623 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:35.619874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.619835 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.643517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.639523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.643811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.643761 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.652621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.652586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:35.674614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.674570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.675399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.675374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:08:35.727513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.727468 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.751468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.729485 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:08:35.886476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.860361 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.912516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.866666 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.946469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.931597 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.969465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.949228 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:35.969950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.969884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.055489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:35.975587 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.063027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.062973 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.094257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.094207 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:36.146470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.135993 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:36.348616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.348083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:36.620529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.595692 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.668473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.635411 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.710479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.691664 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:36.738470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.735840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.738470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.735955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:36.738470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.736318 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.763481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.761507 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.925526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.914426 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:08:36.952469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.927671 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:36.992644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.936113 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.993252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.987364 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.994088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.987485 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:36.994518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:36.987510 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:37.033477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.030585 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.119470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.091225 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.119470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.095156 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.163641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.163584 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.244606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.228994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:37.749639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.749593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.750570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.750540 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": context deadline exceeded" Nov 01 22:08:37.751068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.751046 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:08:37.751493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.751470 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:37.768260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.768222 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.829467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.828605 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.829704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.829651 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.829869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.829836 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.861438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.861398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:37.917580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.917503 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.995805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.995748 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:37.996255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.996228 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:37.998041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.998007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:37.998559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:37.998536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:38.038008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.037961 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.095584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.095534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:08:38.096843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.096810 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:38.164472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.164410 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:38.518906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.518598 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.563406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.563371 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:38.705765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.705732 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:08:38.751072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.751028 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.752548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.752516 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.753166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.752724 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.774970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.774901 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:08:38.776521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.776490 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.850502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.848510 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.850502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.849966 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:38.922526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:38.920725 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:08:39.035550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.034929 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:39.035550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.035241 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:39.044489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.043366 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:39.044489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.043690 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:39.135475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.132651 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:39.135475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.134731 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:39.295648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.295552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:39.382483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.381657 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm"] Nov 01 22:08:39.386479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.382873 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:39.386479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.382904 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:39.386479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.383163 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:39.476758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.476720 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm"] Nov 01 22:08:39.532895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.478355 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.576560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.576256 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:08:39.576961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.576938 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:08:39.630763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.577036 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:08:39.630994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.577633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:39.631233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.578710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.631598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.631566 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jr7l\" (UniqueName: \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.632059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.632038 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.632372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.603656 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:08:39.743132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.743088 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.743549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.743519 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jr7l\" (UniqueName: \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.743895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.743865 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.753472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.745381 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:08:39.753472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.747601 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.756483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.754860 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:39.756483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.755181 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:39.783499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.776233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.795249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.795217 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jr7l\" (UniqueName: \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\") pod \"maas-api-revoke-sub-745e52b1-ltqmm\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:39.822003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.821952 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:39.882525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.872497 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:39.882525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.872883 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:39.972599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:39.972555 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:40.161508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.134424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:40.161508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.135201 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:40.161508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.138171 5036 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": context deadline exceeded" Nov 01 22:08:40.161508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.139157 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:40.161508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.139900 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:08:40.253686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.246402 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:40.253686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.246896 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:40.264481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.264348 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:08:40.335462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.302528 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:40.335462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.304031 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:40.335462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.310808 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:08:40.335462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.329117 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:40.352327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.350584 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:08:40.568041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.565411 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:40.569170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.568684 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:08:40.653907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.653465 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:08:40.658877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.654706 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:08:40.685380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.684161 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:08:40.687528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.686123 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:08:40.701739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.699713 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:08:40.701739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.700550 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:08:40.859275 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:40.859223 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4feac34_c305_47e9_81d8_c36df6d71c20.slice/crio-9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061 WatchSource:0}: Error finding container 9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061: Status 404 returned error can't find the container with id 9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061 Nov 01 22:08:40.873135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:40.872400 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm"] Nov 01 22:08:41.038500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.038457 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:41.530791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.530733 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" event={"ID":"f4feac34-c305-47e9-81d8-c36df6d71c20","Type":"ContainerStarted","Data":"59b765023d34d629e52e5231e4fc7950405fc6bbad911ec63d8c8e387af7d7fd"} Nov 01 22:08:41.530983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.530801 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" event={"ID":"f4feac34-c305-47e9-81d8-c36df6d71c20","Type":"ContainerStarted","Data":"9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061"} Nov 01 22:08:41.555924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.555852 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" podStartSLOduration=3.555834173 podStartE2EDuration="3.555834173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:41.549422998 +0000 UTC m=+1475.242403482" watchObservedRunningTime="2026-11-01 22:08:41.555834173 +0000 UTC m=+1475.248814658" Nov 01 22:08:41.626225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.626189 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:41.646055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.645899 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:41.841287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:41.815313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:42.236208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.236162 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:08:42.237437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.237408 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:42.237583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.237438 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:42.237739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.237702 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9da3113-b386-46e0-9ad1-cd0f3915d26d" containerName="istio-proxy" Nov 01 22:08:42.242337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.242291 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.251644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.251599 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:08:42.266142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266092 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45pp\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266271 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266328 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266497 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266603 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.266754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.266648 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368256 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368316 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368371 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368467 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368505 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368546 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368591 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368630 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.368952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.368791 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45pp\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.372837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.369888 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.372837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.370128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.372837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.370350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.372837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.370600 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.372837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.371818 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.374265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.374226 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.375058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.375034 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.425482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.425431 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45pp\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.429952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.429911 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:42.545461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.545396 5036 generic.go:356] "Generic (PLEG): container finished" podID="f4feac34-c305-47e9-81d8-c36df6d71c20" containerID="59b765023d34d629e52e5231e4fc7950405fc6bbad911ec63d8c8e387af7d7fd" exitCode=0 Nov 01 22:08:42.546874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.546844 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" event={"ID":"f4feac34-c305-47e9-81d8-c36df6d71c20","Type":"ContainerDied","Data":"59b765023d34d629e52e5231e4fc7950405fc6bbad911ec63d8c8e387af7d7fd"} Nov 01 22:08:42.557899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.557874 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:42.619880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:42.619001 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:43.041753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.040487 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:08:43.053227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.053188 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:43.053356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.053282 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:43.053356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.053323 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:43.397664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.397404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:43.567917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.567854 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" event={"ID":"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f","Type":"ContainerStarted","Data":"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4"} Nov 01 22:08:43.567917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.567919 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" event={"ID":"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f","Type":"ContainerStarted","Data":"e19543285aa909bbafde0a047d762cf0ad8f4e80ac6ad5522dcdf6899507307f"} Nov 01 22:08:43.638954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.637858 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:43.847049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.840594 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:08:43.847049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:43.840662 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:44.027470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.025494 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:44.054973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.054889 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:44.057055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.057003 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podStartSLOduration=2.056988126 podStartE2EDuration="2.056988126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:43.611916186 +0000 UTC m=+1477.304896674" watchObservedRunningTime="2026-11-01 22:08:44.056988126 +0000 UTC m=+1477.749968591" Nov 01 22:08:44.087792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.086858 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\") pod \"f4feac34-c305-47e9-81d8-c36df6d71c20\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " Nov 01 22:08:44.087792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.086909 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\") pod \"f4feac34-c305-47e9-81d8-c36df6d71c20\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " Nov 01 22:08:44.087792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.086963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\" (UniqueName: \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\") pod \"f4feac34-c305-47e9-81d8-c36df6d71c20\" (UID: \"f4feac34-c305-47e9-81d8-c36df6d71c20\") " Nov 01 22:08:44.089594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.089536 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca" pod "f4feac34-c305-47e9-81d8-c36df6d71c20" (UID: "f4feac34-c305-47e9-81d8-c36df6d71c20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:08:44.127623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.122042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l" pod "f4feac34-c305-47e9-81d8-c36df6d71c20" (UID: "f4feac34-c305-47e9-81d8-c36df6d71c20"). InnerVolumeSpecName "kube-api-access-7jr7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:08:44.127623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.127504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca" pod "f4feac34-c305-47e9-81d8-c36df6d71c20" (UID: "f4feac34-c305-47e9-81d8-c36df6d71c20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:08:44.204411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.202801 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:44.204411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.202849 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feac34-c305-47e9-81d8-c36df6d71c20-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:44.204411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.202866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jr7l\" (UniqueName: \"kubernetes.io/projected/f4feac34-c305-47e9-81d8-c36df6d71c20-kube-api-access-7jr7l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:08:44.636851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.636733 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:44.637428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.637394 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:44.652469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.652031 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" Nov 01 22:08:44.653362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.652920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm" event={"ID":"f4feac34-c305-47e9-81d8-c36df6d71c20","Type":"ContainerDied","Data":"9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061"} Nov 01 22:08:44.653362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:44.652970 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9187e4062b79ce46dc39b6f5eb51163b9779292e3ae7d038b8da2d596678a061" Nov 01 22:08:45.005094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.005004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:45.322483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.300551 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:08:45.322483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.300656 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:45.621307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.620796 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:45.621307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.620861 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:45.864625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.864579 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:08:45.865124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.864651 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:45.868239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:45.868142 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:46.567906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:46.567861 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:46.619934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:46.619850 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:46.619934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:46.619915 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:47.689424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:47.681149 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:47.689424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:47.683375 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:47.689424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:47.683427 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:48.453648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:48.453606 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:08:48.453897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:48.453672 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:48.623253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:48.622800 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:48.623253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:48.622859 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:48.863141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:08:48.916693 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.136591 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.172098 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.229119 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.368193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.623336 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:49.703297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.623410 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.791523 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.792921 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f4feac34-c305-47e9-81d8-c36df6d71c20" containerName="revoke-keys" Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.792947 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4feac34-c305-47e9-81d8-c36df6d71c20" containerName="revoke-keys" Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.793193 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4feac34-c305-47e9-81d8-c36df6d71c20" containerName="revoke-keys" Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.798583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.907149 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:08:49.912490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:49.907278 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.049639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.019912 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9f8g\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.049639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.020040 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.067618 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.067792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.067829 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.067962 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.067991 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.068016 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.084119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.068052 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.129469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.123593 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dfdb11-openshift-default-dockercfg-8h45l\"" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.172990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.173115 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.173198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.173225 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.180551 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.182345 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.182638 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.223800 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.225965 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.226007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.226033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.226071 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.226244 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9f8g\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.298642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.227025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.320341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.318913 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:50.381471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.380803 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.429569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.429479 5036 generic.go:356] "Generic (PLEG): container finished" podID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerID="e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6" exitCode=0 Nov 01 22:08:50.430995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.430966 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerDied","Data":"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6"} Nov 01 22:08:50.431637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.431617 5036 scope.go:122] "RemoveContainer" containerID="e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6" Nov 01 22:08:50.434949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.434920 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:50.435139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.435118 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:50.435293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.435274 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:50.491743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.490885 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9f8g\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.593058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.563743 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.593058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.568244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.593058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.569792 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv"] Nov 01 22:08:50.608888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.608604 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.616167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.614602 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:08:50.616167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.614762 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:08:50.616167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.615305 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:08:50.649251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.620957 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:08:50.649251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.648007 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv"] Nov 01 22:08:50.649251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.648585 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 01 22:08:50.649251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.648645 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 01 22:08:50.673790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.673594 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:50.710120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.709920 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:50.764526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.763797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.764526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.763841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722vt\" (UniqueName: \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.764526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.764043 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.886587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.874508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.886587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.874553 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722vt\" (UniqueName: \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.886587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.874756 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.886587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.884763 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.901627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.901563 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722vt\" (UniqueName: \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:50.926137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:50.926017 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:51.054827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.052748 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh"] Nov 01 22:08:51.070742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.065280 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4a743-wtssh"] Nov 01 22:08:51.104861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.104632 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:51.151061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.150981 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:51.300516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.296928 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:51.441477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.441381 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:08:51.442550 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:08:51.442507 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec747a1_f425_4e2c_985b_292e1091fb29.slice/crio-a898b2fc6a1e52eb495f9ffb1c17a3748599301d855ebc1e44b09b1488d4e503 WatchSource:0}: Error finding container a898b2fc6a1e52eb495f9ffb1c17a3748599301d855ebc1e44b09b1488d4e503: Status 404 returned error can't find the container with id a898b2fc6a1e52eb495f9ffb1c17a3748599301d855ebc1e44b09b1488d4e503 Nov 01 22:08:51.451278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.451234 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.451397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.451327 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.451397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.451374 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.490181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.490147 5036 generic.go:356] "Generic (PLEG): container finished" podID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerID="91b7c7c3f5965fa88455ea30c62b88fa4e731f1611db7118f5a9860b483ef8bf" exitCode=0 Nov 01 22:08:51.490532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.490505 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerDied","Data":"91b7c7c3f5965fa88455ea30c62b88fa4e731f1611db7118f5a9860b483ef8bf"} Nov 01 22:08:51.493077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.493054 5036 scope.go:122] "RemoveContainer" containerID="91b7c7c3f5965fa88455ea30c62b88fa4e731f1611db7118f5a9860b483ef8bf" Nov 01 22:08:51.495697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.495664 5036 generic.go:356] "Generic (PLEG): container finished" podID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerID="b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1" exitCode=0 Nov 01 22:08:51.497112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.497087 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerDied","Data":"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1"} Nov 01 22:08:51.500384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.500361 5036 scope.go:122] "RemoveContainer" containerID="b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1" Nov 01 22:08:51.559366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.559327 5036 generic.go:356] "Generic (PLEG): container finished" podID="4417b012-afea-41c9-8046-35f17e09053e" containerID="c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c" exitCode=0 Nov 01 22:08:51.559729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.559700 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerDied","Data":"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c"} Nov 01 22:08:51.560368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.560343 5036 scope.go:122] "RemoveContainer" containerID="c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c" Nov 01 22:08:51.564577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.564549 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerStarted","Data":"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5"} Nov 01 22:08:51.572670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.572632 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.572782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.572730 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.572859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.572778 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.574110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.574042 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.574296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.574272 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.574463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.574425 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.582601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.582569 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.582811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.582788 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.583020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.582968 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:08:51.630121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.629864 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:51.631951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.631929 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:51.634865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:51.634846 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:08:52.094793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.092842 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv"] Nov 01 22:08:52.251660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.250263 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:08:52.278605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.278574 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:52.278872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.278841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:52.312064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.288531 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:52.538307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.529880 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4" path="/var/lib/kubelet/pods/ae3c14cb-ce0d-4f87-bfa7-c0b954be71c4/volumes" Nov 01 22:08:52.607053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.607016 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerStarted","Data":"0d9a502f31be338a786c24f1ab65386182ab23afe4aa249971011f5c21e1dc9f"} Nov 01 22:08:52.607270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.607244 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerStarted","Data":"a898b2fc6a1e52eb495f9ffb1c17a3748599301d855ebc1e44b09b1488d4e503"} Nov 01 22:08:52.629364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.629329 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerStarted","Data":"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f"} Nov 01 22:08:52.651747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.651715 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" event={"ID":"788a0592-9c55-468e-bdc1-2951ab49ed55","Type":"ContainerStarted","Data":"d8483f6f1b83302a8a8c0f9ebdeebac0b3ca6fee9ba3ced0db00ad90d7b96bf5"} Nov 01 22:08:52.652541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.645409 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podStartSLOduration=3.645390235 podStartE2EDuration="3.645390235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:52.635880702 +0000 UTC m=+1486.328861189" watchObservedRunningTime="2026-11-01 22:08:52.645390235 +0000 UTC m=+1486.338370722" Nov 01 22:08:52.680520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.678092 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:52.772475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:52.771521 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:08:53.144556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.144525 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:53.279689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.246406 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:53.279689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.246480 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:53.697031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.696733 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:53.733663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.729533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:53.765974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.765908 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" event={"ID":"788a0592-9c55-468e-bdc1-2951ab49ed55","Type":"ContainerStarted","Data":"84292c8fef23d6ab3563b7d3c1838250a44ef6200d0ebc86072488b9634b7edf"} Nov 01 22:08:53.831480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.830572 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" podStartSLOduration=3.830547263 podStartE2EDuration="3.830547263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:53.812407851 +0000 UTC m=+1487.505388338" watchObservedRunningTime="2026-11-01 22:08:53.830547263 +0000 UTC m=+1487.523527776" Nov 01 22:08:53.887797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.867691 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:53.887797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.867765 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:53.939488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.894190 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerStarted","Data":"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb"} Nov 01 22:08:53.986245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:53.986148 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerStarted","Data":"831db4d9842fc91ec52114a6838abe916ea7bdd185a0b00873e46b5179548a94"} Nov 01 22:08:54.019468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.002263 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:54.019468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.003806 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:08:54.019468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.003925 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:08:54.019468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.003980 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:08:54.019468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.015780 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:08:54.276532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.274195 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:54.276532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.274259 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:54.504650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.504611 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:08:54.567559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.566596 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:54.567559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.566661 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:54.725469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.711707 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:54.857560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.801067 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:54.857560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.801792 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:08:54.857560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.801893 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:54.857560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.818009 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:54.857560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.836519 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:54.957297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.955520 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:54.964473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.963609 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:54.992490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.986291 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:08:54.992490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:54.986354 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:08:55.024478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.016729 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.035266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.035230 5036 generic.go:356] "Generic (PLEG): container finished" podID="788a0592-9c55-468e-bdc1-2951ab49ed55" containerID="84292c8fef23d6ab3563b7d3c1838250a44ef6200d0ebc86072488b9634b7edf" exitCode=0 Nov 01 22:08:55.045721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.043313 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" event={"ID":"788a0592-9c55-468e-bdc1-2951ab49ed55","Type":"ContainerDied","Data":"84292c8fef23d6ab3563b7d3c1838250a44ef6200d0ebc86072488b9634b7edf"} Nov 01 22:08:55.128657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.128542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.129483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.128911 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:55.129968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.128946 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.130351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.129364 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.174468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.166258 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.205495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.186581 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.245468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.238540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.340468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.323530 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.386653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.371739 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:55.386653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.371796 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:55.498825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.491026 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:55.498825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.491153 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:55.498825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.493404 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.516492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.513963 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:55.516492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.514112 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.592141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.592083 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.630633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.630591 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.714270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.711933 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:55.714270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.712374 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:55.714270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.712984 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:08:55.787482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.773942 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:55.787482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.774019 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:55.836476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.836204 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.836694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.836591 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:08:55.843752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.842050 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.875719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.875659 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:55.993206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.993117 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:08:55.993473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.993424 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:08:55.994006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:55.993975 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.100468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.043040 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:56.100468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.047616 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:08:56.100468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.047684 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:08:56.133001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.132877 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.135500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.135336 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:08:56.135876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.135849 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.136766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.136366 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:08:56.174924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.174871 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.176261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.176239 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:08:56.187346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.187017 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.191132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.188393 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:08:56.282470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.282334 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:08:56.284398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.284372 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:08:56.315599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.315566 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:56.315871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.315841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:56.465080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.465041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:56.480497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.473318 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:56.480497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.473373 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:56.607525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.603570 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.716158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.708919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.716500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.716439 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:56.716689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:56.716671 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.049521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.038826 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:57.049521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.040020 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.049521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.040269 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:57.050324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.049907 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.050324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.050146 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.091468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.090822 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.111425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.110730 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:08:57.144005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.140593 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:08:57.144005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.140650 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:08:57.144005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.140687 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:08:57.178688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.172566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.178688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.173387 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:57.178688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.173433 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:57.178688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.173511 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.182924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.182516 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:57.185437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.185413 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:08:57.185642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.185621 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:08:57.199860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.199831 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:57.205673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.205627 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.235605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.227929 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.297485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.290800 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:57.297485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.292653 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:57.297485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.292700 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:57.511664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.511568 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:57.780964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:57.774147 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:58.116690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.115215 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.116690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.115780 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:08:58.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.146410 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:58.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.146539 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:58.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.149611 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.222469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.211847 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.250485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.250333 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:08:58.277514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.257935 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:08:58.280004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.279954 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:08:58.310174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.280487 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:08:58.310174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.282482 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:58.310174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.300485 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.310174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.307314 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.310174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.308391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.357052 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.357950 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.374873 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.375285 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.375580 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.379612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.375996 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:08:58.398531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.390687 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:08:58.398531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.390788 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:08:58.398531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.391260 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:58.398531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.391336 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:58.398531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.391641 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:08:58.400632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.399606 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:58.657519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.554603 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 01 22:08:58.737455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.557231 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:08:58.933525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.871572 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:08:58.975468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.954388 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:08:58.997931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:58.977622 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:59.115955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.115916 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:08:59.116252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.116222 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:08:59.117499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.117471 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:59.117716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.117684 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:59.169470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.157775 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.213793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.196806 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:08:59.217466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.214117 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:08:59.235869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.235842 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:08:59.245489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.209368 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:59.268549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.233245 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.282045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.282009 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:08:59.361464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.252226 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.367466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.300605 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.398428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.300819 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:08:59.398771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.302154 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:08:59.398949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.398924 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:08:59.399064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.306964 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:08:59.399283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.399269 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:08:59.442470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.302983 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.663466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.365947 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.663466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.366102 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:08:59.664028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.383700 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:08:59.694572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.383752 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.778274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.383977 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.862991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.325005 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:08:59.867670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.325043 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:08:59.877484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.464592 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:08:59.877821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.877773 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:08:59.877973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.769321 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:08:59.881544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.881519 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:08:59.906466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.713705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\") pod \"788a0592-9c55-468e-bdc1-2951ab49ed55\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " Nov 01 22:08:59.949774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:08:59.949656 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:00.024839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.020880 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:00.024839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.021164 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:00.024839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.022917 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.094660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.081639 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:09:00.094660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.081699 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:09:00.094660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.083167 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.134221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.133706 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.135822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.134598 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:09:00.135822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.134656 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:09:00.135822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.135208 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:00.211609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.208334 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\") pod \"788a0592-9c55-468e-bdc1-2951ab49ed55\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " Nov 01 22:09:00.211609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.208479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\" (UniqueName: \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\") pod \"788a0592-9c55-468e-bdc1-2951ab49ed55\" (UID: \"788a0592-9c55-468e-bdc1-2951ab49ed55\") " Nov 01 22:09:00.276722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.276678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca" pod "788a0592-9c55-468e-bdc1-2951ab49ed55" (UID: "788a0592-9c55-468e-bdc1-2951ab49ed55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:09:00.358847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.309416 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.359176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.309472 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.382899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.382863 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:09:00.411551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.411501 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:09:00.411916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.397651 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.412314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.397600 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:00.439509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.416651 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt" pod "788a0592-9c55-468e-bdc1-2951ab49ed55" (UID: "788a0592-9c55-468e-bdc1-2951ab49ed55"). InnerVolumeSpecName "kube-api-access-722vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:09:00.606663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.440251 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:09:00.634731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.524813 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:09:00.635019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.634985 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:09:00.748141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.748100 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722vt\" (UniqueName: \"kubernetes.io/projected/788a0592-9c55-468e-bdc1-2951ab49ed55-kube-api-access-722vt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:09:01.011151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.826675 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:01.076728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.826897 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:01.077052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.826923 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:01.077366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.848221 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.080504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.848281 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:01.122382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.869366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" event={"ID":"788a0592-9c55-468e-bdc1-2951ab49ed55","Type":"ContainerDied","Data":"d8483f6f1b83302a8a8c0f9ebdeebac0b3ca6fee9ba3ced0db00ad90d7b96bf5"} Nov 01 22:09:01.198466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.179077 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8483f6f1b83302a8a8c0f9ebdeebac0b3ca6fee9ba3ced0db00ad90d7b96bf5" Nov 01 22:09:01.198466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.179284 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:01.198466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.869434 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv" Nov 01 22:09:01.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.884665 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:01.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.885931 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.944369 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.200700 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:09:01.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.023732 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.208503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.886146 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:01.209338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.951336 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.213562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.027705 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.213764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.213729 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:09:01.213891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.162523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.135519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:00.884547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.230281 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.023769 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.232592 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.101539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.239481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.233253 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:01.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.301172 5036 patch_prober.go:29] interesting pod/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.301248 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 22:09:01.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.318390 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca" pod "788a0592-9c55-468e-bdc1-2951ab49ed55" (UID: "788a0592-9c55-468e-bdc1-2951ab49ed55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:09:01.347472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.338347 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a0592-9c55-468e-bdc1-2951ab49ed55-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:09:01.374562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.374508 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:01.403776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.375497 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.429883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.379513 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:09:01.475791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.475595 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:01.482985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.482952 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.483735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.483706 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:09:01.567730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.558995 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:01.810470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.789304 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:01.810470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.789420 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.810470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.789482 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:09:01.990730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.988800 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:09:01.990730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:01.988866 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:09:02.093484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.077538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.113674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.094859 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.137685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.108163 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.137685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.134579 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.254522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.253687 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:02.273475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.271406 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.295467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.290135 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:02.324579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.310502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.324579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.311198 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.324579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.313574 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.364600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.364555 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.381582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.374220 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.397719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.393511 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:02.410289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.402351 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:09:02.431515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.411666 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.431515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.412014 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.450474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.450420 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:02.470361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.470327 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:09:02.470929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.470718 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:09:02.486152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.485525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.670643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.670581 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:02.745511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.712805 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:09:02.745511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.715300 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:02.745511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.725861 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:09:02.815547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.789495 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:09:02.815547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.789558 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 22:09:02.827472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:02.826827 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:03.010517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.003042 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:09:03.010517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.003123 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:09:03.121468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.120583 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.131469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.122668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:03.131469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.123795 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:03.165678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.138011 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.313762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.288054 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.326255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.322543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.385954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.385391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:03.401289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.400672 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.401743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.401703 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:03.402374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.402351 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:03.415203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.415167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.456666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.456614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.460461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.458105 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:03.503096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.503054 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:03.511665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.508505 5036 patch_prober.go:29] interesting pod/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 01 22:09:03.511665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.508562 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 01 22:09:03.548997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.548946 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.550040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.550006 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:03.627615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.609358 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:03.627954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.627932 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:03.628167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.628143 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:03.628349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.628327 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:03.628522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.628507 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:03.699193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.684864 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:03.884922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.884840 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6"] Nov 01 22:09:03.908678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.903301 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:09:03.913460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.910537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:03.924977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.924958 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:09:03.928841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.928824 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:03.935476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.935082 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:03.956381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.956355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:09:03.958408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.957637 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:03.963752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.963727 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb9c4de-5qzs6"] Nov 01 22:09:03.964527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:03.964497 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:04.022004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.021973 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 22:09:04.022248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.022217 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 22:09:04.200476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.200358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:04.484128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.481120 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e92779-2a96-4be8-b04b-75be8d50dd15" path="/var/lib/kubelet/pods/12e92779-2a96-4be8-b04b-75be8d50dd15/volumes" Nov 01 22:09:04.502389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.502355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:09:04.505881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.505838 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:09:04.990230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.990193 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:09:04.991625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:04.991602 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:09:05.039807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:05.039750 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:05.817876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:05.817839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:06.513394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:06.513355 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:07.170035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:07.170002 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:09:07.225063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:07.225024 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:07.324347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:07.315828 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:09:07.988496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:07.984052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:08.650195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.650158 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:08.969677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.969576 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:09:08.970882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.970854 5036 cpu_manager.go:414] "containerMap: removing container" podUID="788a0592-9c55-468e-bdc1-2951ab49ed55" containerName="revoke-keys" Nov 01 22:09:08.970882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.970880 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788a0592-9c55-468e-bdc1-2951ab49ed55" containerName="revoke-keys" Nov 01 22:09:08.971102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.971085 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="788a0592-9c55-468e-bdc1-2951ab49ed55" containerName="revoke-keys" Nov 01 22:09:08.978966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.978944 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:08.986752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.986715 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:09:08.987797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.987773 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f2760d\"/\"openshift-service-ca.crt\"" Nov 01 22:09:08.988667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.988649 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f2760d\"/\"kube-root-ca.crt\"" Nov 01 22:09:08.989488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.989318 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f2760d\"/\"test-model-f2760d-kserve-self-signed-certs\"" Nov 01 22:09:08.989985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:08.989423 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f2760d\"/\"default-dockercfg-fj86f\"" Nov 01 22:09:09.061727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.061432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:09.075989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.075952 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gxt\" (UniqueName: \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.076119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.076085 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.076174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.076134 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.076245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.076229 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.076347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.076332 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.076672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.076652 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180494 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180656 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gxt\" (UniqueName: \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180755 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180786 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.187470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.180873 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.191583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.191473 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.192374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.192318 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.192823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.192802 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.224504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.220753 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.247463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.233702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gxt\" (UniqueName: \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.263881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.263780 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\") pod \"test-model-f2760d-kserve-74f8b46877-gcmrb\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.488543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.474428 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:09.712264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.712092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:09.834662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:09.834349 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:09:10.200739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.200666 5036 scope.go:122] "RemoveContainer" containerID="37b9a1a54d3c0b37bc0b6155e769a6b238be8a432c215eff9efee6ad51cb23dc" Nov 01 22:09:10.444931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.444895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:10.491792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.491689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" event={"ID":"c194ac77-f017-48ed-916f-bd7425f9dcb9","Type":"ContainerStarted","Data":"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b"} Nov 01 22:09:10.491792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.491748 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" event={"ID":"c194ac77-f017-48ed-916f-bd7425f9dcb9","Type":"ContainerStarted","Data":"e881e97bbe630b84e61b629faa5ce0fa9f3ebd472a780ad1e36c4d93e312be64"} Nov 01 22:09:10.513989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.513929 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podStartSLOduration=2.513911098 podStartE2EDuration="2.513911098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:10.512439309 +0000 UTC m=+1504.205419795" watchObservedRunningTime="2026-11-01 22:09:10.513911098 +0000 UTC m=+1504.206891585" Nov 01 22:09:10.683146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.683097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:10.822283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:10.822219 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:11.475279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:11.475244 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:11.626674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:11.626635 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:12.513051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:12.513017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:13.259611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:13.259572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:14.128237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:14.128197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:15.202438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:15.202396 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:15.611557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:15.611514 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:16.020436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:16.020404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:16.177220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:16.177179 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:16.561470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:16.561291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:17.249133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:17.249101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:17.389467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:17.381172 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:18.161504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:18.161467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:18.821012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:18.820982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:18.910297 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:09:18.910252 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:09:19.362125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.362093 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:09:19.423045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.411655 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:09:19.423045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.411781 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.445076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.445036 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8483c2\"/\"default-dockercfg-tl4bt\"" Nov 01 22:09:19.445076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.445055 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8483c2\"/\"kube-root-ca.crt\"" Nov 01 22:09:19.445076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.445055 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8483c2\"/\"openshift-service-ca.crt\"" Nov 01 22:09:19.445369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.445106 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8483c2\"/\"test-model-8483c2-kserve-self-signed-certs\"" Nov 01 22:09:19.456431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.456396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.456636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.456607 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.456717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.456660 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.456892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.456870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.456955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.456913 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.457095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.457069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99pdj\" (UniqueName: \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.474780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.474710 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:19.492465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.489431 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:19.561716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.561675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645405 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645497 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645699 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645866 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99pdj\" (UniqueName: \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.645940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.668464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.658268 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.668464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.658632 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.678467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.675257 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.678467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.675870 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.701470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.699998 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.757474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.750880 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99pdj\" (UniqueName: \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\") pod \"test-model-8483c2-kserve-b9f44f459-r6w8b\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:19.881507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:19.873870 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:20.043415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.043380 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:20.168680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.152145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:20.435594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.434725 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:20.494302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.494263 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:09:20.561866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.561834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:20.732405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.732324 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" event={"ID":"b3b2c441-fb50-46bd-ae3b-a0f346b033e2","Type":"ContainerStarted","Data":"e0340cfb2998d47aaa88f9ab3ba5837df6d6ede7ee55ea97164ce88220578177"} Nov 01 22:09:20.732405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.732374 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" event={"ID":"b3b2c441-fb50-46bd-ae3b-a0f346b033e2","Type":"ContainerStarted","Data":"248affcba3d308578fe0d67352723f3a0b27a226e7ceac8be55a773fee9a1922"} Nov 01 22:09:20.760701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:20.760643 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podStartSLOduration=1.760613881 podStartE2EDuration="1.760613881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:20.754604316 +0000 UTC m=+1514.447584802" watchObservedRunningTime="2026-11-01 22:09:20.760613881 +0000 UTC m=+1514.453594368" Nov 01 22:09:21.043667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:21.043637 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:21.165776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:21.165738 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:21.627506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:21.626759 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:22.049524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:21.983548 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:22.730925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:22.730886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:23.595850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:23.595810 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:24.325047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:24.325011 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:25.038594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:25.038561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:25.947969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:25.947933 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:26.794899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:26.794839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:28.547569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:28.547534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:29.342621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:29.342565 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:29.450651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:29.450594 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:29.610520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:29.609613 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:29.721942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:29.721888 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:29.949785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:29.949669 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.032140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.032093 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.043689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.043643 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:30.196277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.194361 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:30.207902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.207799 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:30.209576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.209535 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.289586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.289529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:30.335235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.335185 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:30.348107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.348053 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.380480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.369385 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:30.382188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.382154 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:30.382709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.382680 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.410634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.410584 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.639501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.626382 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:30.677475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.671542 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.737623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.737572 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:30.881488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.874189 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:30.881488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.874265 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:30.968179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.968074 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.983272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.968708 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.983272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:30.971648 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:31.015228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.014711 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.151467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.122122 5036 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.208009 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.211584 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.212041 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.213140 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.213524 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.214502 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:31.228571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.214805 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:31.305948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.305903 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:31.358304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.358263 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:31.362613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.361911 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.393276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.367133 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:31.393878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.371382 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.392542 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.398617 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.392786 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.392817 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.399852 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:09:31.400476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.399901 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:31.428126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.428023 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.448148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.428527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.527525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.482571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:31.527525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.521699 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.681982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.681542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.753746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.753482 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 22:09:31.767895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.766594 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.976935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.976841 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:31.989479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.983794 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:32.028483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:31.990210 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:32.086238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.030070 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.086238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.063589 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:32.086238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.063653 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:32.156337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.111552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.262580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.250098 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:32.262580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.251963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:32.262580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.253118 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:32.269484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.269434 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:32.269774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.269732 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.380504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.374738 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9sjh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:32.380504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.374805 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.395700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.395659 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.418313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.396814 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.418313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.398049 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:32.418313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.400204 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.418854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.418799 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:32.441777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.441720 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.676835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.455893 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:32.678296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.455927 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:32.700070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.537947 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.715543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.549704 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.759208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.759167 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.924462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.684527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:32.978657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:32.977592 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.116159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.116116 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.116658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.116505 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.132469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.125362 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:33.132469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.125428 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:33.132469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.125500 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:09:33.150765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.149535 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.197469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.191533 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:33.612487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.604006 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:33.612487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.606242 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.620475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.614623 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:33.654576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.647661 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.660954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.660912 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.666177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.666145 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:33.681701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.681650 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.695495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.686536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:33.695750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.695710 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.717998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.717963 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.740528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.720191 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.807189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.807143 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:33.855520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:33.821621 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:34.128080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.128042 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:34.213471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.158624 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.213471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.196249 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 01 22:09:34.213471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.127143 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.213471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.200513 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.250259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.250198 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:34.335472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.318221 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:34.401153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.395738 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:34.550996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.550958 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:34.606540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.606401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.640492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.640455 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:34.641286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.641252 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.647541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.647512 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.667939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.667859 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.706392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.687101 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:34.729740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.729697 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:34.846565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.779562 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:35.164480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.164427 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:35.164668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.164520 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:35.164668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.164562 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:35.164668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.164601 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:35.179468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:34.876669 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.179468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.167097 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:35.179468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.013200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:35.190872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.190844 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:35.205341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.205304 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.225619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.225248 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.287859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.287826 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:35.299592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.299560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:35.301584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.301525 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.479652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.479540 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/health\": context deadline exceeded" Nov 01 22:09:35.574472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.574270 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.575301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.575273 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:35.613941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.612641 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:35.647034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.646992 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:35.674617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.674589 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:35.687014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.686984 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:35.687268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.687253 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:35.708138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.708100 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:35.714048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.714027 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:35.753649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.741151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.806469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.802609 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:35.853121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:35.853090 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:36.036994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.036949 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:36.165531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.164315 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:36.194472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.194421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:36.418687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.418580 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:36.895485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.723401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:36.895485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.758666 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:36.895485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.758705 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:36.923511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.810784 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:36.923511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:36.815747 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.060469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.049993 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.060469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.050733 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:37.134155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.134112 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.207818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.207517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:37.258466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.229779 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:37.306661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.306629 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:37.374218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.233486 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:37.374555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.276188 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.375032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.232770 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.375733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.319867 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:37.376126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.320051 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:37.499523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.477053 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:09:37.600678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.600630 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.683469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.680221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:37.683469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.680401 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.683469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.680827 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:37.683469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.681215 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:37.756596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:37.753247 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:38.054472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.029763 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.225510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.116655 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:38.381780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.125620 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:38.385507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.202007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:38.435469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.202285 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 22:09:38.435469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.251459 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.436941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.256638 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:38.437100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.265048 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.445469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.374866 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.445666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.382047 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.455539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.382101 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:38.513494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.382124 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:38.513494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.445723 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:38.542468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.541947 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.682652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.682542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.740117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.740067 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:38.841475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.841202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:38.887667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.887625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:38.924410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.924372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:38.952141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.952050 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Nov 01 22:09:39.066571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:38.997767 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.109477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.109338 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.126562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.126523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136090 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136153 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136197 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136238 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136276 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136313 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136346 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136380 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:39.139247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.136416 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:39.214869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.209135 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.214869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.209828 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.381488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.376541 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.386621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.386042 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:39.493590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.489746 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:39.493590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.490055 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.495075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.494840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.510074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.510023 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.534503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.515800 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.534503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.516085 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:39.534503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.516324 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:39.578807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.578769 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:39.769704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.716755 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:39.915472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.902600 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:39.973269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.973227 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:40.062524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:39.929050 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:40.062524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.046141 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.063556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.063512 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.288779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.063957 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:40.314665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.314581 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:40.314925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.314909 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:09:40.315059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.315048 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.315167 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.127249 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.221686 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.142738 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.317105 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.331126 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:40.349881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.345125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:40.397555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.397510 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:40.414471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.406230 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:40.414471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.414243 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:40.436483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.423714 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:40.436483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.435743 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:40.515572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.514518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.639755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.639601 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:40.739778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.720778 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:40.763078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.763046 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:40.763383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.763366 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:40.763552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.763541 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:40.764584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.764101 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:40.764584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.764185 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:40.764584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.764331 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:40.764960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.764939 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:40.765056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.765038 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:40.765126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.765114 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:40.767124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.766293 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:40.891687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.889463 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:09:40.895966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.895939 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:40.901920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.901384 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:09:40.908014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.907993 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:09:40.908323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.908308 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:40.917913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:40.917887 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.027173 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028158 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028243 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028551 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mjd\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028607 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028643 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csp8p\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028707 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028772 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.028924 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.029145 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.029834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.029945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.030463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.029979 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.036301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.030174 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.036301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.030654 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.036301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.030808 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.132760 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.132891 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.132937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mjd\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.132979 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csp8p\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133106 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133186 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133287 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133389 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133432 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133484 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133528 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133568 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133600 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133787 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.133851 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.134211 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.134937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.134572 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.136318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.135141 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.136318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.135660 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.136318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.135765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.136318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.136088 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.136604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.136566 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.138020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.137468 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.141558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.141499 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.144782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.142490 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.145272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.145249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.150498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.150439 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.154911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.154886 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.160434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.160389 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mjd\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.162509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.162321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csp8p\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.163340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.163025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.164760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.164680 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.230071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.230038 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:41.247488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.245760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:41.337491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.337439 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:41.493183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.492467 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:09:41.500240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.500143 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.514390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.508788 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:09:41.521891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.521366 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.527334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.527177 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:09:41.544537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.544296 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:09:41.552891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.552866 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:09:41.630614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.630578 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:09:41.631029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.630918 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.642313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.635877 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-830149-openshift-default-dockercfg-kkt65\"" Nov 01 22:09:41.651531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651501 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651566 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651608 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651671 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msrj\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651737 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651813 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651838 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.651901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651871 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651955 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.651996 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.652018 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6gn\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.652075 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.652123 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.652280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.652143 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.678081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.678048 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.754827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.754890 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.754932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.754963 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755002 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755044 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9gg\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755182 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755240 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755264 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6gn\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755288 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755319 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755377 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755420 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755482 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755530 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755686 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755794 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755853 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.755982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.756023 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msrj\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.756100 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.756131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.757376 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.757762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.758050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.758613 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.759147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.759308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.759778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.759840 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.760061 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.760116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.763533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.760738 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.771829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.765350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.771829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.765418 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.771829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.766529 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.773493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.773025 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6gn\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.776365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.776332 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.780188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.780158 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msrj\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.781924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.781862 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.796846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.796472 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:09:41.804832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.804783 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.805153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.805056 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.805310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.805128 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.812934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.812788 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:09:41.814821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.814792 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.814924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.814858 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.814924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.814901 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:41.824902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.824872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:41.859321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859290 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.859321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859339 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.859698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.859698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859511 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.859698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859654 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.859698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859680 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.860136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.860136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.859937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.860136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.860015 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9gg\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.863835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.863004 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.863835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.863268 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.865272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.864642 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.865272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.864902 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.865272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.864907 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.865272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.865237 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.870753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.870728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.882218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.882188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9gg\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.884789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.884763 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:41.930428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.930200 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:41.972157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.971837 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:41.976936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:41.976706 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:42.257325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.257188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:42.355501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.348844 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" event={"ID":"57f901f6-dc11-49bf-acdc-72cae5c2c9f7","Type":"ContainerStarted","Data":"d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0"} Nov 01 22:09:42.355501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.348930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" event={"ID":"57f901f6-dc11-49bf-acdc-72cae5c2c9f7","Type":"ContainerStarted","Data":"da67fdb44603759318437f354891b512f429f36354dc8b2fadc3dec9377f9752"} Nov 01 22:09:42.355501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.352157 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" event={"ID":"68b7ab59-79fa-4c9f-8356-bcbb1118eb20","Type":"ContainerStarted","Data":"4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b"} Nov 01 22:09:42.355501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.352211 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" event={"ID":"68b7ab59-79fa-4c9f-8356-bcbb1118eb20","Type":"ContainerStarted","Data":"8eded9dc560d9812a4ba767b44dff6c69f9438c4aeacdcdd504c9aecec1b519b"} Nov 01 22:09:42.379349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.379291 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podStartSLOduration=2.37927147 podStartE2EDuration="2.37927147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:42.37378672 +0000 UTC m=+1536.066767207" watchObservedRunningTime="2026-11-01 22:09:42.37927147 +0000 UTC m=+1536.072251957" Nov 01 22:09:42.469365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.467715 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podStartSLOduration=2.46769567 podStartE2EDuration="2.46769567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:42.460831736 +0000 UTC m=+1536.153812223" watchObservedRunningTime="2026-11-01 22:09:42.46769567 +0000 UTC m=+1536.160676157" Nov 01 22:09:42.639508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.621289 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:09:42.639508 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:09:42.634143 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b6afa6_2296_4256_9802_3bdc72f29da7.slice/crio-b6531d1f2ab0e26e546a53a41b508d43e804261530f2b403172dab9158cb7908 WatchSource:0}: Error finding container b6531d1f2ab0e26e546a53a41b508d43e804261530f2b403172dab9158cb7908: Status 404 returned error can't find the container with id b6531d1f2ab0e26e546a53a41b508d43e804261530f2b403172dab9158cb7908 Nov 01 22:09:42.739468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.722371 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.739468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.722494 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.739468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.722537 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.834581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.834479 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.834724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.834587 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.834724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.834630 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.951745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.951703 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:09:42.965251 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:09:42.965157 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eac855d_ad84_4737_b0ad_bf2dea25eefd.slice/crio-d097ccdd80712da8dedb01de7510e9241996d468f3f78b8451c6b83d91ec5757 WatchSource:0}: Error finding container d097ccdd80712da8dedb01de7510e9241996d468f3f78b8451c6b83d91ec5757: Status 404 returned error can't find the container with id d097ccdd80712da8dedb01de7510e9241996d468f3f78b8451c6b83d91ec5757 Nov 01 22:09:42.996153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.984611 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.996153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.984712 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:42.996153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:42.984758 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:09:43.040475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.038994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:43.170666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.170617 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:09:43.250793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.250752 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:43.262039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.260089 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:43.262039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.260140 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:43.262039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.260157 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:43.262039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.260197 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:43.262039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.260380 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:43.530385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.530340 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" event={"ID":"f8b6afa6-2296-4256-9802-3bdc72f29da7","Type":"ContainerStarted","Data":"263480c0f00375b79cc7941c9e23c4ec57c8f72ad08b00877a26c18e3e2afc3f"} Nov 01 22:09:43.569968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.568736 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podStartSLOduration=2.568715064 podStartE2EDuration="2.568715064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:43.551612904 +0000 UTC m=+1537.244593394" watchObservedRunningTime="2026-11-01 22:09:43.568715064 +0000 UTC m=+1537.261695551" Nov 01 22:09:43.575202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.575168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" event={"ID":"f8b6afa6-2296-4256-9802-3bdc72f29da7","Type":"ContainerStarted","Data":"b6531d1f2ab0e26e546a53a41b508d43e804261530f2b403172dab9158cb7908"} Nov 01 22:09:43.575617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.575578 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" event={"ID":"8eac855d-ad84-4737-b0ad-bf2dea25eefd","Type":"ContainerStarted","Data":"d097ccdd80712da8dedb01de7510e9241996d468f3f78b8451c6b83d91ec5757"} Nov 01 22:09:43.601915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.601869 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" event={"ID":"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49","Type":"ContainerStarted","Data":"57c6776153b11ce4f89e3b11f63fcfb9347604a1b14eec9bf952d6b8b50f3dd3"} Nov 01 22:09:43.979851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:43.979769 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:44.074506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.074374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:44.273503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.273473 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:44.273815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.273767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:44.274340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.274318 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:44.274528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.274490 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:44.507469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.505534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:44.591145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.591041 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:44.673324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.673284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" event={"ID":"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49","Type":"ContainerStarted","Data":"ef9c3df2af2986ef8a79709c0f177029e1e1c54e0e34a72e7f281afcfc6c3ed0"} Nov 01 22:09:44.678587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.678554 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" event={"ID":"8eac855d-ad84-4737-b0ad-bf2dea25eefd","Type":"ContainerStarted","Data":"837fcc815486c7bdfac99e5fcaa503333072ff0f86410fe5ea300510fe18b831"} Nov 01 22:09:44.774465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.752505 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:44.774465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.763038 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:44.774465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.766528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:44.774465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.767424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:44.821817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.821550 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:44.961943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.945270 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:44.979467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.976037 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:44.979467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.976153 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:44.979467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:44.976191 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:45.007618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.007580 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:45.007871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.007843 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:45.127481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.116010 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podStartSLOduration=4.115988923 podStartE2EDuration="4.115988923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:44.752181429 +0000 UTC m=+1538.445161917" watchObservedRunningTime="2026-11-01 22:09:45.115988923 +0000 UTC m=+1538.808969410" Nov 01 22:09:45.222530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.147926 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podStartSLOduration=4.147898186 podStartE2EDuration="4.147898186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:44.803916408 +0000 UTC m=+1538.496896896" watchObservedRunningTime="2026-11-01 22:09:45.147898186 +0000 UTC m=+1538.840878673" Nov 01 22:09:45.222530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.169524 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:45.226461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.225391 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.250043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.248560 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:45.250043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.248620 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:45.250581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.250479 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:45.250581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.250530 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:45.287337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.278831 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.306943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.305972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.325436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.323600 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.325436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.323839 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:45.325436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.324189 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:45.409746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.409689 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.435674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.435425 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.483682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.483570 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.515703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.514690 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.566743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.563600 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.566743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.564300 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:45.612381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.609559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.612381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.610174 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:45.792528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.772917 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.792528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.773318 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:45.821921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.821871 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:45.824332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.824306 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:45.878554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.871363 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:45.906540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.879494 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.906540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.880582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:45.906540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.880783 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:45.977522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.973522 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:09:45.977522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:45.973629 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:46.130373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.130219 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:46.357081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.357029 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:46.374324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.374290 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:46.474673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.444833 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:46.500593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.487012 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:46.500593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.489679 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:46.500593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.489730 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:46.542477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.534877 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.010763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.616347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:47.035808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.650496 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:47.037280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.664676 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.056497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.666247 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.058160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.679106 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.058390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.058360 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:47.058545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.692668 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.058860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.679200 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.202540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.720741 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.203638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.720773 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.204955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.768822 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:47.206637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.768940 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:47.207787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.799957 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:47.209159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.799995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:47.213724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.884659 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.213724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.741677 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.213724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.213153 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:47.213724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.138421 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": context deadline exceeded" Nov 01 22:09:47.213724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.151163 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:47.214104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.720660 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.217566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.748104 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.217733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.217708 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:47.218129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.844614 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.218298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.218274 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:47.218976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:46.846136 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.219164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.219127 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:47.238615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.238583 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.306471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.244238 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.306471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.297479 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:47.306471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.244664 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.306471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.298733 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:47.307657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.307617 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.307852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.307831 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:47.312522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.309037 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.313677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.297750 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:47.357840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.357644 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.478809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.478587 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.493205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.492563 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.576792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.576702 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.577053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.577023 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:47.665154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.659965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:47.714645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.704666 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:47.827553 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:09:47.827465 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.381s" Nov 01 22:09:47.835599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.835560 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:47.847697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.847662 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:47.859609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.859576 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:47.870295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.859837 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:47.870646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.870622 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:47.883601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.883568 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:47.908673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.908642 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:47.908915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:47.908900 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:48.013610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.013558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.108841 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.112028 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.154624 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.558666 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.391608 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.391577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.489587 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.559003 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.522047 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.522075 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.392346 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.397958 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.520876 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.496467 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.559758 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.547105 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.522096 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.522124 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.522141 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.530534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.547061 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.496639 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.560646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.496678 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.561498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.560691 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:48.612079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.563604 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.612079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.564434 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:48.612079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.584977 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:48.612079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.597107 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.690386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.630546 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:48.991526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.940526 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.991815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:48.991775 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:49.075697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.045658 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:49.153508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.153454 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:49.153958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.117844 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:49.154152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.154124 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:49.155034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.155010 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:49.155286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.155268 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:49.155465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.155435 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:49.155620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.155608 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:09:49.170284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.170255 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:49.309264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.309227 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:49.445552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.420050 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:49.709358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.709265 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.733560 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.733600 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.835372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.836527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.837739 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.840025 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.841347 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.841895 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:49.842469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.842147 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:49.846847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.846814 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:49.848879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.792637 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:49.849656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.849634 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:49.883098 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:09:49.864078 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:09:49.883098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.734436 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:50.020191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.004588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:50.020191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.018898 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:50.032469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.815575 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:50.034897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.815612 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:50.035621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.815633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:50.035994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.815656 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:50.037115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.815674 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:50.052634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.827459 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.052915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.052879 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:50.053071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:49.928964 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:50.053202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.053183 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.053420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.053395 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.067105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.067071 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.067410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.067380 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:50.076585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.067631 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.077154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.077121 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:50.407905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.246011 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:50.408186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.307105 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.408430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.408403 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:50.408593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.307280 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.408706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.408686 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:50.408819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.397791 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:50.408929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.408910 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:50.526706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.526664 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.748778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.748681 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 01 22:09:50.778131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.778093 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:50.795597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.795556 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.832163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.828778 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:50.848598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.848546 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.858539 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.858548 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.858925 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.858938 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.859189 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:50.877470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.859433 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:50.891494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.879755 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:50.892051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.882552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.902486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.900956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:50.906661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.906613 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:50.929040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.928618 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.005659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:50.999989 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:51.005659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.000056 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:51.008864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.008835 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:51.009106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.009081 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:51.022221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.020787 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.069465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.041722 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.093251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.093209 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:51.107573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.103957 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:51.166589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.123017 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:51.166589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.123206 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:51.166589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.123252 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:51.166589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.128748 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.171668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.171577 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.247475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.234550 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:51.247475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.234609 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:51.278531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.274002 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:51.278531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.274054 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:51.859740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.535430 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:51.880528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.864097 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.912477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.897845 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.912477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.898255 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:51.921471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.912837 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.921471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.913168 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:51.921471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.914232 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:51.921471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.916084 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.923832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.923804 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:51.987545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.924522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:51.987545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.924569 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:51.987545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.924605 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:51.992485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.953882 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:51.992485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:51.953965 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.066661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.066625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:52.078088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.076017 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:52.078088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.076090 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:52.078088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.077435 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:52.078088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.077492 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:52.079150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.079117 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:52.079289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.079175 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:52.079496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.079466 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.247365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.096988 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.259467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.097034 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.259467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.102720 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:52.259814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.125662 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.260279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.129380 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.290487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.285584 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:52.290487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.285640 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:52.290487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.285754 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:52.290487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.285776 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:52.602502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.567102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:52.605144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.603923 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.892605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.889654 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.918466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.906905 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:52.918466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:52.907411 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.118931 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.119234 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.146035 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.146089 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.146172 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.146193 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.165830 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.165891 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.167311 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.167624 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.167915 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.168149 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.168374 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.168637 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.168900 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:53.210554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.169211 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:53.222418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.221640 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:53.250590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.234890 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.250590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.234944 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:53.250590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.235433 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:53.250590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.248020 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.250590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.248072 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:53.253524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.251716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:53.253524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.252093 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:53.289036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.270989 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:53.289036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.271523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:53.289036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.271790 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:53.289036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.272007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:53.670607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.670556 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:53.795022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.794968 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:53.827798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.827666 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 01 22:09:53.943468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.911684 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:53.943468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.912080 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:53.943468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.942933 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.943468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.942986 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:53.989818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.989031 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.989818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.989101 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:53.989818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.989717 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:53.989818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:53.989760 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:54.230530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.201850 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:54.230530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.202230 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:54.230530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.202543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:54.230530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.202789 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:09:54.230530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.203086 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:54.263830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.260708 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:54.263830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.260789 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:54.356755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.356709 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 01 22:09:54.384991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.357037 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:54.385202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.357068 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:54.385427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.357088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.360224 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.360257 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.360276 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.360294 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.360312 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.175290 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.373544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:54.421471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.373575 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:54.451939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.451888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:54.608915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.490180 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:54.608915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.606052 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:54.608915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.544534 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:54.608915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.606437 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:54.746328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.746274 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:54.835519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.835467 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:54.943532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.918367 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:55.010862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.962559 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:55.019563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.019516 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:55.020106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:54.962599 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:55.100469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.088016 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:55.100469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.088088 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:55.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.126504 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:55.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.126662 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:55.154466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.126706 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:55.293592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.285650 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:09:55.293592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.285710 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 01 22:09:55.293592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.286955 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:55.296015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.295969 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.296473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.296271 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:09:55.315476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.314192 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:09:55.315476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.314242 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 01 22:09:55.315476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.314996 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:55.363878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.353253 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.363878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.360144 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.391055 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.391337 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.391551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.391792 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.391952 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:55.392835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.392100 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:55.397388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.393427 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.397388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.393669 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:55.397388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.393833 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:55.432469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.400882 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:55.432469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.428321 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.453500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.452520 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/health\": context deadline exceeded" Nov 01 22:09:55.550523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.548291 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:55.550523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.548356 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.613608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.612618 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:55.613608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.613305 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:55.665070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.665034 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:09:55.665830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.665806 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:09:55.788468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.773985 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:55.907294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.891316 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": context deadline exceeded" Nov 01 22:09:55.981763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:55.981728 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:56.148486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.127573 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:56.148486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.071986 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:56.148486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.051895 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:09:56.155869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.072017 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:56.156244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.070403 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:56.157247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.156156 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:56.157247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.072715 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:56.157247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.156437 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:56.307477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.306662 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:09:56.307477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.306720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:56.309329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.309299 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:56.385471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.352018 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:56.518759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.352661 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:56.598808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.353285 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:56.624467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.353422 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:09:56.624761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.425868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:09:56.649713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.425908 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:56.653466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.425928 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:56.678759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.425948 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:09:56.679130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.425967 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:56.704469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.429410 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:56.706488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.454629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:09:56.737200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.454675 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:09:56.738177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.454695 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:09:56.738384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.562492 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:56.765487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.744536 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:56.775532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:56.774573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:57.053111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.045552 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:57.083613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.083582 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:57.393003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.388320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:57.393003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.388366 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:57.468405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.467791 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:09:57.469208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.468677 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:09:57.521708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.521648 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:09:57.554887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.552815 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:09:57.587474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.587405 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:09:57.651714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.094365 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:57.651996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.651963 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:57.674475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.126796 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:57.750331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.750291 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:57.772465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.702291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:57.774821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.126838 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:09:57.776217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.776170 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 01 22:09:57.859166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859128 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:57.859373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859241 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:09:57.859373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859303 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:09:57.859373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859358 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:09:57.859686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859402 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:09:57.859686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859472 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:09:57.859686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:57.859534 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:09:58.012475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.000633 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 01 22:09:58.012475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.000696 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 01 22:09:58.209303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.207919 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:58.209303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.208310 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:58.256366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.230617 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:09:58.256366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.230966 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:09:58.256366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.231294 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:09:58.256366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.231544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:09:58.267653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.267627 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:09:58.286853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.286804 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:58.320572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.230525 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": context deadline exceeded" Nov 01 22:09:58.322076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.322035 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:58.340886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.340859 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:09:58.341394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.337528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:58.343368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.343341 5036 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:09:58.343556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.343530 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 01 22:09:58.373509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.368709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:09:58.373509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.369073 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:09:58.376997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.376510 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:58.455403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.455355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:09:58.516641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.516562 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:09:58.524472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.517402 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:09:58.524735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.524715 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:09:58.562817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.562788 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:09:58.565637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.565617 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:09:58.571154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.571130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:09:58.836413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.836383 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:58.975844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.975758 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:09:58.987703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:58.987411 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:59.185549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:59.174827 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:59.244591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:59.244101 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:09:59.257699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:59.257666 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:09:59.263419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:59.263389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:09:59.824010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:09:59.819280 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:00.525506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:00.525408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:01.093240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:01.093198 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:10:01.098007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:01.096610 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:01.329487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:01.329415 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:02.222041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:02.221990 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:02.848691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:02.779435 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:02.958282 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.101480 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.272421 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv"] Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.458372 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0f9b7e-77pgv"] Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.468070 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:03.883432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.704681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:03.945669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.942416 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:04.034432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.953778 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": read tcp 10.132.0.2:55502->10.132.0.209:15021: read: connection reset by peer" start-of-body= Nov 01 22:10:04.034432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:03.953843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": read tcp 10.132.0.2:55502->10.132.0.209:15021: read: connection reset by peer" Nov 01 22:10:04.087487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.085507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:04.199107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.197796 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 22:10:04.199107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.198061 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" containerID="cri-o://7c1e2d899f42ed385097e16fec88287eea8c5aa00192ad3c6bb291e665796fc0" gracePeriod=30 Nov 01 22:10:04.359978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.343171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:04.461131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.461100 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerID="61211256e61c64af7bee645462b01d381845ef03a2bc55062bef8d6d065f6a25" exitCode=0 Nov 01 22:10:04.464513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.464397 5036 generic.go:356] "Generic (PLEG): container finished" podID="a40b676b-5ad7-423d-9223-abed4d646fee" containerID="1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368" exitCode=0 Nov 01 22:10:04.467584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.467558 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788a0592-9c55-468e-bdc1-2951ab49ed55" path="/var/lib/kubelet/pods/788a0592-9c55-468e-bdc1-2951ab49ed55/volumes" Nov 01 22:10:04.468085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.468061 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerDied","Data":"61211256e61c64af7bee645462b01d381845ef03a2bc55062bef8d6d065f6a25"} Nov 01 22:10:04.468168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.468128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerDied","Data":"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368"} Nov 01 22:10:04.469160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.469134 5036 scope.go:122] "RemoveContainer" containerID="1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368" Nov 01 22:10:04.469357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.469333 5036 scope.go:122] "RemoveContainer" containerID="61211256e61c64af7bee645462b01d381845ef03a2bc55062bef8d6d065f6a25" Nov 01 22:10:04.472538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.472503 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:04.472717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.472694 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:04.472788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.472753 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:04.473514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.473285 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:04.473514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.473363 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:04.473514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:04.473397 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:05.038572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.038531 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:05.478625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.478481 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerStarted","Data":"18f1eed6d67cbc7705b015cbc7f6bd9e2506993f0f41d0c8537a50704306fdef"} Nov 01 22:10:05.481238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.481204 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerStarted","Data":"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d"} Nov 01 22:10:05.483618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.483592 5036 generic.go:356] "Generic (PLEG): container finished" podID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerID="28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c" exitCode=0 Nov 01 22:10:05.483767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.483670 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerDied","Data":"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c"} Nov 01 22:10:05.484040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.484020 5036 scope.go:122] "RemoveContainer" containerID="28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c" Nov 01 22:10:05.485960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.485933 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:05.486052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.486003 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:05.486052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.486037 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:05.492386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.492365 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:10:05.493047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.493026 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:05.493149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.493071 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:05.753007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:05.752917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:06.264539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.264499 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:06.264984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.264569 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:06.264984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.264969 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:10:06.497467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.496850 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:06.497467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.496913 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:06.565777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.565713 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerStarted","Data":"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264"} Nov 01 22:10:06.692155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.692109 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:10:06.716358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.714890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:06.716358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.716350 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:06.716648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:06.716402 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:07.270590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.265858 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:07.270590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.265934 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:07.447465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.438338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:07.493465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.492633 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:07.493465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.492693 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:07.695920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.695827 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:07.696234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:07.696199 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:08.236035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.235999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:08.311592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.311545 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:08.313601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.313572 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:08.313812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.313789 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:08.581504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.559555 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:08.581504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.576526 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:08.624864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.624818 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:08.626077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.626052 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:08.635200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.635166 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:08.698375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.698327 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:08.698948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.698906 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:08.770503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.768131 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:08.857533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.845999 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:10:08.857533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:08.846415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:10:09.030471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.024602 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:10:09.055478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.051202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:09.107481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.100173 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:09.296502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.295686 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:09.296502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.295749 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:09.423169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.423129 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:10:09.495834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.495735 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:09.496583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.496074 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:09.563513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.562073 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:09.563513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.563219 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:10:09.591614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.591533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:09.593570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.593517 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:10:09.621598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.621395 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:09.639486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.635884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:09.897090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:09.897049 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.018361 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.019993 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.021336 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.021381 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.039522 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.041158 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:10:10.070651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.041504 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:10:10.112718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.112677 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:10:10.127806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.117591 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:10:10.146536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.146123 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:10.200915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.200824 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:10:10.201574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.201519 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" containerID="cri-o://831db4d9842fc91ec52114a6838abe916ea7bdd185a0b00873e46b5179548a94" gracePeriod=30 Nov 01 22:10:10.310041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.307331 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.310041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.307559 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:10:10.310041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.307997 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.371807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.327619 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.441484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.382566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:10:10.441484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.392507 5036 scope.go:122] "RemoveContainer" containerID="2a438d18d8f2ac4a336b758931be4373719d22e057d4898780c6df255572988a" Nov 01 22:10:10.491587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.464811 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:10.503658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.503608 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:10.504034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.503918 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:10.589275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.588227 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.610779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.610737 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.662821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.662786 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:10.663132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.663103 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:10.663245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.637236 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:10:10.672554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.611889 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.687209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.603010 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:10.687769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.637285 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 01 22:10:10.793598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.769199 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:10.793598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.769273 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:10.793598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.792807 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:10:10.794492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.794071 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:10:10.794492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.794152 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:10:10.826013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.825979 5036 generic.go:356] "Generic (PLEG): container finished" podID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerID="7c1e2d899f42ed385097e16fec88287eea8c5aa00192ad3c6bb291e665796fc0" exitCode=0 Nov 01 22:10:10.827423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.826298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" event={"ID":"03bae2a6-5630-4751-82bc-6ae358c9e943","Type":"ContainerDied","Data":"7c1e2d899f42ed385097e16fec88287eea8c5aa00192ad3c6bb291e665796fc0"} Nov 01 22:10:10.900503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.900461 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:10:10.920302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.911960 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:10:10.920302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:10.918607 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:10:11.075520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.033169 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:10:11.352417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.352280 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:11.352717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.352668 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:11.355257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.100596 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.376600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.100666 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:10:11.384406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.384352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.384901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.124933 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.391902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.227727 5036 scope.go:122] "RemoveContainer" containerID="18696a4b9ec23bcbf93084fa726e8131867c4264f92f2dbd1f901fff8da0c51c" Nov 01 22:10:11.392148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.341588 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.405959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.405936 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:10:11.414958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.414938 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:10:11.511298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.049727 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:10:11.512614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.049770 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.516490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.243678 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.516490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.456715 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:10:11.516490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.456747 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:10:11.516490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.512876 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.517032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.456795 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:10:11.517032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.456769 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:10:11.517032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.504333 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:11.517032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.516980 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:11.587557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.563692 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:10:11.587557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.563746 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:10:11.587557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.563800 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:10:11.591472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.590649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.591472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.591017 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:10:11.613235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.610233 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr"] Nov 01 22:10:11.675196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.655796 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:10:11.675196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.673063 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.728368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.728241 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:11.750553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.750490 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.779480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.732519 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:11.779480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.754139 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.779480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.754168 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:10:11.844613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.840522 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:11.894815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.894619 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr"] Nov 01 22:10:11.895359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.895129 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:11.922455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.919764 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:10:11.922455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.920275 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:10:11.934509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.933382 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:10:11.968506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.950264 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:10:11.968506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.950391 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:10:11.968506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:11.950517 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:10:12.101749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.099654 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:12.163979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.157846 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:10:12.374764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.287690 5036 scope.go:122] "RemoveContainer" containerID="f616d092845004ec7f1d4a0afe6dc115d0caacb16810edb1924552c596209965" Nov 01 22:10:12.459789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.451013 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:12.459789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.451151 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ll5\" (UniqueName: \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:12.459789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.451676 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:12.544224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.484747 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:10:12.544224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.492649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:10:12.544224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.518995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:10:12.544224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.520994 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:10:12.671595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.626689 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:12.671595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.626767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.708584 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.709212 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.709255 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.710491 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.710986 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.711218 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:10:12.725804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.711708 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:12.734009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.731414 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:10:12.734009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.733877 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:12.734009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.733933 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:12.806571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.793219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.225776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ll5\" (UniqueName: \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.226422 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.803605 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.938725 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.921681 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.922590 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:10:13.227630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.938855 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.939236 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.939039 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.961189 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.961406 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.023788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.001151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.055321 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.103364 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:10:13.269469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:12.960535 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:10:13.309074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.309024 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:13.419471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.400636 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:13.625500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.624111 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:10:13.625500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.624195 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:10:13.659507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.659360 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:10:13.733467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.724317 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.771633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.743576 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ll5\" (UniqueName: \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:13.771894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.749988 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.772143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.750088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:10:13.772345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.749941 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:10:13.772568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.750567 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:10:13.772789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.750698 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:10:13.772983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.750726 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Nov 01 22:10:13.773187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.750739 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:13.874932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.872695 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:14.233471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.896642 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 01 22:10:14.272292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.272252 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 01 22:10:14.272292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.905173 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:10:14.272572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.272369 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:10:14.272572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.272398 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" event={"ID":"03bae2a6-5630-4751-82bc-6ae358c9e943","Type":"ContainerDied","Data":"81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf"} Nov 01 22:10:14.272572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.272472 5036 scope.go:122] "RemoveContainer" containerID="7c1e2d899f42ed385097e16fec88287eea8c5aa00192ad3c6bb291e665796fc0" Nov 01 22:10:14.272746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.191641 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.272746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.244238 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.272892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.897910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:14.357565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.346558 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:10:14.357565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.346884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:10:14.357565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.244533 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.357565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.347415 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:10:14.377480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.271684 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:10:14.377480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.371781 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:10:14.377480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.371833 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 01 22:10:14.377480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.271320 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:10:14.377480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.271653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.955266 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411016 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.955527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.951778 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411141 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411335 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411362 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411543 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411707 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.411851 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.412949 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.434470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:13.958559 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng" Nov 01 22:10:14.471683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.438279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:14.471683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.445135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.502915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.502875 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:14.540523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.150508 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 01 22:10:14.568894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.552104 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:10:14.568894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.552241 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:14.584474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.244285 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.584720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.244305 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.591656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.587304 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "kube-api-access-559zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:14.621650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.620228 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.631216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.631166 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.638859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.638827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.675465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.659217 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:14.800716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.786835 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 01 22:10:14.800716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.786899 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 01 22:10:14.800716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.799700 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 22:10:14.830719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.820182 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:14.834253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.834224 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.834701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.834683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.834947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.834929 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.835061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.835048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.835208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.835196 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.835322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.835309 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") pod \"03bae2a6-5630-4751-82bc-6ae358c9e943\" (UID: \"03bae2a6-5630-4751-82bc-6ae358c9e943\") " Nov 01 22:10:14.836505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.836430 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bae2a6-5630-4751-82bc-6ae358c9e943-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.836634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.836622 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.839279 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839242 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~empty-dir/istio-envoy Nov 01 22:10:14.839279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839266 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.839475 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839318 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~projected/kube-api-access-559zc Nov 01 22:10:14.839475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839382 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "kube-api-access-559zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:14.839475 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839437 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~downward-api/istio-podinfo Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839536 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~projected/istio-token Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839566 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839613 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~empty-dir/workload-socket Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839622 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:14.839659 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes/kubernetes.io~empty-dir/istio-data Nov 01 22:10:14.839703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.839667 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data" pod "03bae2a6-5630-4751-82bc-6ae358c9e943" (UID: "03bae2a6-5630-4751-82bc-6ae358c9e943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:14.966604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.966524 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559zc\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-kube-api-access-559zc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.966948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.966862 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.967114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.967100 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.967272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.967257 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.967425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.967410 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:14.968686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:14.968665 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bae2a6-5630-4751-82bc-6ae358c9e943-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.025914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.025876 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:10:15.028854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.027788 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:10:15.028854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.027882 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:10:15.028854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.027910 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:10:15.029962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.029923 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" Nov 01 22:10:15.030100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.029964 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" Nov 01 22:10:15.033136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.030572 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" containerName="istio-proxy" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041634 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041690 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041724 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041755 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041785 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041814 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041844 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041878 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041910 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041940 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041968 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.041998 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.042029 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.042066 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:10:15.045474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.042158 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.069901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.069844 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:10:15.125934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.121664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:15.167171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.167131 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:10:15.227895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.227788 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.236260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.236229 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.236552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.236531 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9mp\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.236712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.236696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.236829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.236814 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.237018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.237005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.237128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.237116 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.237223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.237211 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.237551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.237529 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.342257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.342202 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.342648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.342627 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.342942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.342925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343100 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9mp\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343258 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343398 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343571 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343738 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.343901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.343886 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.344684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.344665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.347300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.347112 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.351061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.347671 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.351280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.348115 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.351405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.348887 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.355729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.355692 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:10:15.371424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.371389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:10:15.396690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.396656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9mp\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.399929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.399902 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:10:15.413275 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:10:15.401411 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bae2a6_5630_4751_82bc_6ae358c9e943.slice/crio-81bdab95391d277f694896422e17d3f723bbf5a2b369dec5fb6a1be70eada6cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bae2a6_5630_4751_82bc_6ae358c9e943.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:10:15.437306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.436905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.455282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.455231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.462365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.460822 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.470718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.470684 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 22:10:15.473651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.473628 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:15.483215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.483143 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gczng"] Nov 01 22:10:15.523316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.523281 5036 generic.go:356] "Generic (PLEG): container finished" podID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerID="831db4d9842fc91ec52114a6838abe916ea7bdd185a0b00873e46b5179548a94" exitCode=0 Nov 01 22:10:15.526667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.526634 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerDied","Data":"831db4d9842fc91ec52114a6838abe916ea7bdd185a0b00873e46b5179548a94"} Nov 01 22:10:15.526806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.526703 5036 scope.go:122] "RemoveContainer" containerID="91b7c7c3f5965fa88455ea30c62b88fa4e731f1611db7118f5a9860b483ef8bf" Nov 01 22:10:15.580127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.580081 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr"] Nov 01 22:10:15.723050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.721533 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:10:15.725957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.725834 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:10:15.766571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.766053 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:10:15.875258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875217 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875426 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875541 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875611 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875767 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875822 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.875936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.875922 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\") pod \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\" (UID: \"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc\") " Nov 01 22:10:15.879345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.879077 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:15.879345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.879159 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:15.879345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.879217 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:15.879665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.879514 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:15.879795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.879774 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:15.884300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.884259 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:10:15.884429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.884389 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:15.885997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.885962 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "kube-api-access-vskmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:15.911816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.908181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy" pod "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" (UID: "edce6c5a-6cbf-4471-b32d-b98c8f6b33cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:15.916984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.916952 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:15.917105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.917063 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:15.917171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.917116 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:15.924597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.924573 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.949198 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950559 5036 cpu_manager.go:414] "containerMap: removing container" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950581 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950596 5036 cpu_manager.go:414] "containerMap: removing container" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950605 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950840 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.952784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.950854 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" containerName="istio-proxy" Nov 01 22:10:15.966694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.966281 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:15.966694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.966519 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:10:15.979684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979594 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979621 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vskmg\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-kube-api-access-vskmg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979638 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979652 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979705 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979721 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979746 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979761 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.979990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.979776 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:15.986397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:15.986366 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-070e3d-openshift-default-dockercfg-qwkpd\"" Nov 01 22:10:16.038924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.038701 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:16.081345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.081321 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.081672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.081652 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.081913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.081894 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsp2l\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.082185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.082163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.082534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.082516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.082720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.082703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.082854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.082842 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.083052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.083039 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.083186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.083171 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.190466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.186822 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.204764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.204713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.204965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.204825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.204965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.204863 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.204965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.204955 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.205145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.204981 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.205145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.205101 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.205272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.205172 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.205272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.205233 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsp2l\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.205776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.191351 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.208156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.207148 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.208156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.207423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.208156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.207694 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.208934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.208906 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.214146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.212734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.214146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.213495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.239815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.239707 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsp2l\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.240159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.240113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.323654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.323612 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:16.453359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.452632 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bae2a6-5630-4751-82bc-6ae358c9e943" path="/var/lib/kubelet/pods/03bae2a6-5630-4751-82bc-6ae358c9e943/volumes" Nov 01 22:10:16.547712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.547686 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" Nov 01 22:10:16.548060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.547712 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv" event={"ID":"edce6c5a-6cbf-4471-b32d-b98c8f6b33cc","Type":"ContainerDied","Data":"ad80d5d290e535ee530de366eac3cb5331c3a659400778adf95164b5ec16aff1"} Nov 01 22:10:16.548176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.548133 5036 scope.go:122] "RemoveContainer" containerID="831db4d9842fc91ec52114a6838abe916ea7bdd185a0b00873e46b5179548a94" Nov 01 22:10:16.551508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.551327 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" event={"ID":"f8d8f8ce-292d-41e1-bb53-bba9251b5e22","Type":"ContainerStarted","Data":"3637b4e39eef984ad5ba3af99accd974ec95f294b9bead8b7c6dbf90b890a4b0"} Nov 01 22:10:16.553345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.553306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" event={"ID":"f8d8f8ce-292d-41e1-bb53-bba9251b5e22","Type":"ContainerStarted","Data":"fec62b5ac0a1f33474ee6d402ea888ed901d124f742c76f5c84031e922f8c4db"} Nov 01 22:10:16.553859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.553833 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" event={"ID":"5d581ae5-b177-4a34-b112-f612ba985df8","Type":"ContainerStarted","Data":"6c0b36972c4c85dab0bb1aeaf2ddf66f90020229c096968a79d624c4a8bc845e"} Nov 01 22:10:16.554019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.553998 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" event={"ID":"5d581ae5-b177-4a34-b112-f612ba985df8","Type":"ContainerStarted","Data":"964141e254aa37199e40c3b2f5cee6cd7d57c48ee4c899be4c3ca293cd522c53"} Nov 01 22:10:16.574281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.574200 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" podStartSLOduration=6.574187724 podStartE2EDuration="6.574187724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:16.571859341 +0000 UTC m=+1570.264839828" watchObservedRunningTime="2026-11-01 22:10:16.574187724 +0000 UTC m=+1570.267168210" Nov 01 22:10:16.599595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.599541 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podStartSLOduration=35.59952375 podStartE2EDuration="35.59952375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:16.595902902 +0000 UTC m=+1570.288883389" watchObservedRunningTime="2026-11-01 22:10:16.59952375 +0000 UTC m=+1570.292504238" Nov 01 22:10:16.621599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.621425 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:10:16.629273 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:16.629237 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e31560_3854_46f3_b8bb_aa04b5325eb9.slice/crio-23e35d206c419d195690a7b56ce78531d9fce9707295d51b903f09c25bd30abe WatchSource:0}: Error finding container 23e35d206c419d195690a7b56ce78531d9fce9707295d51b903f09c25bd30abe: Status 404 returned error can't find the container with id 23e35d206c419d195690a7b56ce78531d9fce9707295d51b903f09c25bd30abe Nov 01 22:10:16.630130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.630110 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0f9b7e-openshift-default-b776cc9f-z7gbv"] Nov 01 22:10:16.632774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.632741 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:16.632866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.632815 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:16.632941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.632864 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:16.635831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.635804 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:10:16.731094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:16.731063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:17.500523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.499893 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:17.512034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.511997 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:17.512309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.512281 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:17.546346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.546319 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:17.642406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.641164 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerStarted","Data":"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488"} Nov 01 22:10:17.642406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.641228 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerStarted","Data":"23e35d206c419d195690a7b56ce78531d9fce9707295d51b903f09c25bd30abe"} Nov 01 22:10:17.646976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.646307 5036 generic.go:356] "Generic (PLEG): container finished" podID="f8d8f8ce-292d-41e1-bb53-bba9251b5e22" containerID="3637b4e39eef984ad5ba3af99accd974ec95f294b9bead8b7c6dbf90b890a4b0" exitCode=0 Nov 01 22:10:17.646976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.646899 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" event={"ID":"f8d8f8ce-292d-41e1-bb53-bba9251b5e22","Type":"ContainerDied","Data":"3637b4e39eef984ad5ba3af99accd974ec95f294b9bead8b7c6dbf90b890a4b0"} Nov 01 22:10:17.696494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:17.677018 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podStartSLOduration=31.676997079 podStartE2EDuration="31.676997079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:17.668000186 +0000 UTC m=+1571.360980672" watchObservedRunningTime="2026-11-01 22:10:17.676997079 +0000 UTC m=+1571.369977566" Nov 01 22:10:18.323987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.323951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:18.331796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.331758 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:18.404085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.404054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:18.475739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.475702 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edce6c5a-6cbf-4471-b32d-b98c8f6b33cc" path="/var/lib/kubelet/pods/edce6c5a-6cbf-4471-b32d-b98c8f6b33cc/volumes" Nov 01 22:10:18.477664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.476499 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:18.477664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.476551 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:18.661882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.661795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:18.662934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:18.662911 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:10:18.937229 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:10:18.937050 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:10:19.122339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.122311 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:19.227521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.223755 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:19.282469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.274275 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\" (UniqueName: \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\") pod \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " Nov 01 22:10:19.282469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.274390 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\") pod \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " Nov 01 22:10:19.282469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.274519 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\") pod \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\" (UID: \"f8d8f8ce-292d-41e1-bb53-bba9251b5e22\") " Nov 01 22:10:19.303504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.299067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca" pod "f8d8f8ce-292d-41e1-bb53-bba9251b5e22" (UID: "f8d8f8ce-292d-41e1-bb53-bba9251b5e22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:19.359186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.359126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5" pod "f8d8f8ce-292d-41e1-bb53-bba9251b5e22" (UID: "f8d8f8ce-292d-41e1-bb53-bba9251b5e22"). InnerVolumeSpecName "kube-api-access-56ll5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:19.375596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.363001 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca" pod "f8d8f8ce-292d-41e1-bb53-bba9251b5e22" (UID: "f8d8f8ce-292d-41e1-bb53-bba9251b5e22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:10:19.420136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.377579 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ll5\" (UniqueName: \"kubernetes.io/projected/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-kube-api-access-56ll5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:19.420136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.377614 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:19.420136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.377632 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8f8ce-292d-41e1-bb53-bba9251b5e22-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:19.476514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.476325 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:19.476514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.476379 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:19.672460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.672414 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" event={"ID":"f8d8f8ce-292d-41e1-bb53-bba9251b5e22","Type":"ContainerDied","Data":"fec62b5ac0a1f33474ee6d402ea888ed901d124f742c76f5c84031e922f8c4db"} Nov 01 22:10:19.672635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.672487 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec62b5ac0a1f33474ee6d402ea888ed901d124f742c76f5c84031e922f8c4db" Nov 01 22:10:19.672635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.672517 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr" Nov 01 22:10:19.919157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:19.919120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:20.055517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:20.055468 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:20.240004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:20.239966 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:20.476784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:20.476695 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:20.476784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:20.476767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:20.947145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:20.943544 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:21.029390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.028121 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:10:21.123628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.119169 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:21.496196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.494997 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:21.496196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.495084 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:21.601471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.564341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:21.869476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.866202 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:10:21.919695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.919666 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f8d8f8ce-292d-41e1-bb53-bba9251b5e22" containerName="revoke-keys" Nov 01 22:10:21.919912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.919898 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d8f8ce-292d-41e1-bb53-bba9251b5e22" containerName="revoke-keys" Nov 01 22:10:21.921992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.921962 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d8f8ce-292d-41e1-bb53-bba9251b5e22" containerName="revoke-keys" Nov 01 22:10:21.935754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:21.932014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:22.470558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.462726 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:10:22.470558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.463232 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.470558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.463547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:10:22.470558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.469750 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:10:22.478466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.477947 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:22.478466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.478000 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:22.492601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.490295 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:10:22.511181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.495868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.511181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.496923 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.522223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.522189 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.523165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.523135 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.641464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.624881 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:10:22.669507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.663261 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:10:22.669507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.664256 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:22.675541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.675514 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-070e3d\"" Nov 01 22:10:22.695356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.695330 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-070e3d-dockercfg-r2cl6\"" Nov 01 22:10:22.736542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.721325 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:22.871437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.871389 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ftz\" (UniqueName: \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:22.871636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.871561 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:22.973604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.973569 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ftz\" (UniqueName: \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:22.973978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.973958 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:22.974680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:22.974658 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:23.029519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.028269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ftz\" (UniqueName: \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\") pod \"payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:23.029730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.029702 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:23.414613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.414576 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:10:23.417290 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:23.417256 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5092adf_ec66_4db1_b343_ef465a163e8c.slice/crio-9f4f96892822ddb9fa5bbdc59bb5a9fc90f6817a9eaf61fee18607cf2448888a WatchSource:0}: Error finding container 9f4f96892822ddb9fa5bbdc59bb5a9fc90f6817a9eaf61fee18607cf2448888a: Status 404 returned error can't find the container with id 9f4f96892822ddb9fa5bbdc59bb5a9fc90f6817a9eaf61fee18607cf2448888a Nov 01 22:10:23.429669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.427202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:23.476572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.476545 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 01 22:10:23.476717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.476610 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 01 22:10:23.560691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:23.537467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" event={"ID":"a5092adf-ec66-4db1-b343-ef465a163e8c","Type":"ContainerStarted","Data":"9f4f96892822ddb9fa5bbdc59bb5a9fc90f6817a9eaf61fee18607cf2448888a"} Nov 01 22:10:24.514470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:24.513925 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:24.702468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:24.700168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" event={"ID":"a5092adf-ec66-4db1-b343-ef465a163e8c","Type":"ContainerStarted","Data":"1d3fd27013759c6c48b3c4c97733c15b283c65b49bfa45cc1e1f28dbcd83449e"} Nov 01 22:10:24.702468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:24.700868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:24.748223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:24.748150 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" podStartSLOduration=3.74813005 podStartE2EDuration="3.74813005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:24.747058549 +0000 UTC m=+1578.440039035" watchObservedRunningTime="2026-11-01 22:10:24.74813005 +0000 UTC m=+1578.441110540" Nov 01 22:10:25.098477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:25.096521 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:25.098477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:25.098426 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:25.100676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:25.100649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:10:25.398245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:25.398132 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:26.127996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:26.127958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:27.023131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:27.018863 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:27.835273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:27.835233 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:28.508513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:28.508467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:29.251207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:29.251173 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:30.028334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:30.028297 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:30.688857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:30.688821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:31.433201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:31.433103 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:32.118336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:32.118300 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:32.757461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:32.757407 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:33.448925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:33.448871 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:34.104958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:34.104924 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 22:10:34.109524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:34.109486 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" containerID="cri-o://35f7ff00745ee5ce8ab52c0ffbe57d9e3a429e202d8599e0befad084612e09ea" gracePeriod=30 Nov 01 22:10:34.159646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:34.159610 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:34.844738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:34.844686 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:35.026769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.026732 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:35.044015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.043946 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947"] Nov 01 22:10:35.053888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.053777 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a8505-wz947"] Nov 01 22:10:35.722343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.722254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:35.760522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.759649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:10:35.905614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.905577 5036 generic.go:356] "Generic (PLEG): container finished" podID="cec747a1-f425-4e2c-985b-292e1091fb29" containerID="0d9a502f31be338a786c24f1ab65386182ab23afe4aa249971011f5c21e1dc9f" exitCode=0 Nov 01 22:10:35.906147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.905681 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerDied","Data":"0d9a502f31be338a786c24f1ab65386182ab23afe4aa249971011f5c21e1dc9f"} Nov 01 22:10:35.906226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.906149 5036 scope.go:122] "RemoveContainer" containerID="0d9a502f31be338a786c24f1ab65386182ab23afe4aa249971011f5c21e1dc9f" Nov 01 22:10:35.908631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.908595 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:35.908767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.908671 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:35.908767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:35.908702 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:36.444948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:36.444864 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ead712b-a22d-4252-a516-bd6792e4de42" path="/var/lib/kubelet/pods/4ead712b-a22d-4252-a516-bd6792e4de42/volumes" Nov 01 22:10:36.524694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:36.524658 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:36.919785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:36.919751 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerStarted","Data":"6c50ff9f619edd413d0829c5633e6961d60cbcaf3cecc054611702c5f0509ef6"} Nov 01 22:10:37.203688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:37.203594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:37.674088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:37.674056 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:10:37.675871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:37.675841 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:37.676027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:37.675900 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:38.013535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:38.013115 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:38.711470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:38.696562 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:38.711470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:38.696633 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:38.740346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:38.740313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:39.008526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.002522 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:10:39.424437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.424388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:39.583498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.580411 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:39.698520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.697707 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:39.698520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.697764 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:39.902665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:39.889001 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:40.422266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.422218 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:40.596537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.596494 5036 generic.go:356] "Generic (PLEG): container finished" podID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerID="35f7ff00745ee5ce8ab52c0ffbe57d9e3a429e202d8599e0befad084612e09ea" exitCode=0 Nov 01 22:10:40.658826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.658790 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" event={"ID":"2493eb28-61df-4874-a82b-d2ce69e3f1bd","Type":"ContainerDied","Data":"35f7ff00745ee5ce8ab52c0ffbe57d9e3a429e202d8599e0befad084612e09ea"} Nov 01 22:10:40.674836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.674777 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:40.674967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.674838 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:40.736294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.736263 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 22:10:40.820485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.804502 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:10:40.820485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.805872 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" Nov 01 22:10:40.820485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.805897 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" Nov 01 22:10:40.834196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.821096 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" containerName="istio-proxy" Nov 01 22:10:40.834361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.834270 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.838063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.838035 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:10:40.858589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.858559 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.858754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.858652 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.858754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.858697 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.858896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.858808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.859037 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.859212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.859562 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.859890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.859972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860013 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860034 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:40.860436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860051 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\") pod \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\" (UID: \"2493eb28-61df-4874-a82b-d2ce69e3f1bd\") " Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860677 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860712 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860805 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860922 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860975 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:40.861153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.860999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861423 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhwz\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861484 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861639 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861655 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861674 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861688 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.861714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.861703 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.865961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.865833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:10:40.865961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.865937 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:10:40.867662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.867641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "kube-api-access-c4vpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:40.867800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.867749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token" pod "2493eb28-61df-4874-a82b-d2ce69e3f1bd" (UID: "2493eb28-61df-4874-a82b-d2ce69e3f1bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:40.963535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963411 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhwz\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963514 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963734 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963765 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963893 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963930 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.963988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.963954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964127 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964249 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964332 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964401 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964421 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vpd\" (UniqueName: \"kubernetes.io/projected/2493eb28-61df-4874-a82b-d2ce69e3f1bd-kube-api-access-c4vpd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964437 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964500 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2493eb28-61df-4874-a82b-d2ce69e3f1bd-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:40.964581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964522 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.964860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.964834 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.966513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.966492 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.966598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.966586 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.976069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.976047 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhwz\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:40.976187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:40.976167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:41.156318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.156279 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:41.406166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.363749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:41.546870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.545651 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:10:41.547698 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:10:41.547671 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b6e3b1_0b63_4544_8c85_553a784350ee.slice/crio-92ce322e2bc39e7686ab2f2eb061bdb25889ae9792978fd0fccb1359d0277e11 WatchSource:0}: Error finding container 92ce322e2bc39e7686ab2f2eb061bdb25889ae9792978fd0fccb1359d0277e11: Status 404 returned error can't find the container with id 92ce322e2bc39e7686ab2f2eb061bdb25889ae9792978fd0fccb1359d0277e11 Nov 01 22:10:41.551334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.551300 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:41.551496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.551370 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:41.551496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.551405 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:10:41.635994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.635807 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" event={"ID":"2493eb28-61df-4874-a82b-d2ce69e3f1bd","Type":"ContainerDied","Data":"7efa3f4ac5be6717aa54950f367ad6aec493f33cdf8457d4cda86a47547aa829"} Nov 01 22:10:41.636670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.636481 5036 scope.go:122] "RemoveContainer" containerID="35f7ff00745ee5ce8ab52c0ffbe57d9e3a429e202d8599e0befad084612e09ea" Nov 01 22:10:41.636818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.636799 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d" Nov 01 22:10:41.640106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.640041 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" event={"ID":"54b6e3b1-0b63-4544-8c85-553a784350ee","Type":"ContainerStarted","Data":"92ce322e2bc39e7686ab2f2eb061bdb25889ae9792978fd0fccb1359d0277e11"} Nov 01 22:10:41.674461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.674307 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:41.674461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.674371 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:41.738304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.738266 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 22:10:41.751949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:41.751877 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-8dx9d"] Nov 01 22:10:42.071620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.070611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:42.447290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.447215 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2493eb28-61df-4874-a82b-d2ce69e3f1bd" path="/var/lib/kubelet/pods/2493eb28-61df-4874-a82b-d2ce69e3f1bd/volumes" Nov 01 22:10:42.652144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.652107 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" event={"ID":"54b6e3b1-0b63-4544-8c85-553a784350ee","Type":"ContainerStarted","Data":"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82"} Nov 01 22:10:42.674155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.674128 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 01 22:10:42.674311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.674188 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 01 22:10:42.712519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.712391 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podStartSLOduration=16.71237147 podStartE2EDuration="16.71237147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:42.67642115 +0000 UTC m=+1596.369401640" watchObservedRunningTime="2026-11-01 22:10:42.71237147 +0000 UTC m=+1596.405351957" Nov 01 22:10:42.753574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:42.753530 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:43.157489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.157409 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:43.158671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.158644 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:43.158799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.158697 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:43.434976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.434886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:43.690521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.684718 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:10:43.700740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:43.697075 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:10:44.129636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:44.129421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:44.157886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:44.157849 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:44.158281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:44.158234 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:44.936390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:44.936351 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:45.157012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:45.156974 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:45.157218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:45.157032 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:45.634890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:45.634852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:46.157269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:46.157232 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:46.157684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:46.157289 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:46.311428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:46.311386 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:46.999025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:46.998950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:47.156648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:47.156616 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:47.156841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:47.156671 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:47.756945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:47.756912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:48.156791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:48.156757 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:10:48.156977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:48.156809 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 01 22:10:48.636954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:48.636916 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:48.860656 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:10:48.860616 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:10:49.162530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:49.162501 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:49.162888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:49.162866 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:49.163533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:49.163516 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:10:49.323745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:49.323707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:49.984282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:49.984237 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:50.208464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.208412 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:50.344383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.344301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:50.418613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.418575 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr"] Nov 01 22:10:50.713527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.713425 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2a91d-2v2cr"] Nov 01 22:10:50.764993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.764944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:50.918349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:50.918250 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:51.777423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:51.777378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:52.443213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:52.443181 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d8f8ce-292d-41e1-bb53-bba9251b5e22" path="/var/lib/kubelet/pods/f8d8f8ce-292d-41e1-bb53-bba9251b5e22/volumes" Nov 01 22:10:52.520569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:52.520537 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:53.570370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:53.570333 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:54.035906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:54.035864 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:10:54.640743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:54.640687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:55.378058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:55.378014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:56.429002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:56.428962 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:57.125678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:57.125637 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:57.437092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:57.437009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:57.794601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:57.794557 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:57.924530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:57.924492 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:58.079540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:58.079436 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:10:58.080414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:58.080373 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" containerID="cri-o://18f1eed6d67cbc7705b015cbc7f6bd9e2506993f0f41d0c8537a50704306fdef" gracePeriod=30 Nov 01 22:10:58.136513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:58.136458 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:58.779927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:58.779801 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:58.928493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:58.920279 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:10:59.050916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.050830 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:10:59.051241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.051219 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.092551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.091786 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-487639\"/\"openshift-service-ca.crt\"" Nov 01 22:10:59.092551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.092531 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-487639\"/\"test-model-487639-kserve-self-signed-certs\"" Nov 01 22:10:59.147528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.147361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:59.171643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.171609 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-487639\"/\"kube-root-ca.crt\"" Nov 01 22:10:59.172183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.172156 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-487639\"/\"default-dockercfg-wwn5z\"" Nov 01 22:10:59.180520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.180047 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.180911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.180881 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.181053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.181034 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pdx\" (UniqueName: \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.181229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.181213 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.181461 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.181626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.181611 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297275 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pdx\" (UniqueName: \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297582 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.297749 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.298225 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.311484 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.311746 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.311945 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.320573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.313557 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:59.333252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.332478 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.357478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.342350 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pdx\" (UniqueName: \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\") pod \"test-model-487639-kserve-5fbcd86d85-xms2k\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:10:59.473466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.472853 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:10:59.544165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:10:59.542843 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:00.169472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.162091 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:00.359858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.359758 5036 patch_prober.go:29] interesting pod/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:11:00.359858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.359843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:11:00.385466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.384552 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:00.539071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.539041 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:11:00.924601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:00.924494 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:01.355680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:01.355638 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" event={"ID":"6c98d127-a563-4526-ba06-1c87eac0fb85","Type":"ContainerStarted","Data":"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273"} Nov 01 22:11:01.356195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:01.355697 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" event={"ID":"6c98d127-a563-4526-ba06-1c87eac0fb85","Type":"ContainerStarted","Data":"8a02ec640c1664b70c8aa19c6b854a5ff365bc89dee5443ed101c8df5b6c71ac"} Nov 01 22:11:01.379702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:01.379642 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podStartSLOduration=3.379622441 podStartE2EDuration="3.379622441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:11:01.376836195 +0000 UTC m=+1615.069816686" watchObservedRunningTime="2026-11-01 22:11:01.379622441 +0000 UTC m=+1615.072602928" Nov 01 22:11:01.547466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:01.543489 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:01.694561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:01.688196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:02.416530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:02.416493 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:03.125907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.125876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:03.397553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.390746 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerID="18f1eed6d67cbc7705b015cbc7f6bd9e2506993f0f41d0c8537a50704306fdef" exitCode=0 Nov 01 22:11:03.397553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.391217 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerDied","Data":"18f1eed6d67cbc7705b015cbc7f6bd9e2506993f0f41d0c8537a50704306fdef"} Nov 01 22:11:03.397553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.391300 5036 scope.go:122] "RemoveContainer" containerID="61211256e61c64af7bee645462b01d381845ef03a2bc55062bef8d6d065f6a25" Nov 01 22:11:03.442432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.442408 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:11:03.481557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.481517 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:11:03.482757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.482731 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:03.482942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.482919 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:03.483549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.483523 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:03.483549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.483549 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:03.483786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.483773 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:03.488766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.488744 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.519480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.519428 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:11:03.584239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584338 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584579 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584708 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584720 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:03.584751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584738 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584764 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584873 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.584994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:03.584994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.584986 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\") pod \"6f0217d3-06f8-425d-9978-1a3b7d817686\" (UID: \"6f0217d3-06f8-425d-9978-1a3b7d817686\") " Nov 01 22:11:03.585521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.585428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.585756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.585732 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.586422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.585959 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:03.586422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.586010 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:03.586422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.586057 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.586713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.586512 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.586774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.586730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tzt\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.586877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.586855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.587282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.587262 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:03.587513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.587495 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:11:03.587677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.587655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.587783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.587720 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.587783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.587751 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.588108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588084 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588113 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0217d3-06f8-425d-9978-1a3b7d817686-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588129 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588140 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588147 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588177 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588193 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0217d3-06f8-425d-9978-1a3b7d817686-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.588469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.588248 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:11:03.593892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.593867 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc" pod "6f0217d3-06f8-425d-9978-1a3b7d817686" (UID: "6f0217d3-06f8-425d-9978-1a3b7d817686"). InnerVolumeSpecName "kube-api-access-9mmqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:11:03.689502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689387 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tzt\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689841 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689853 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.689929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.689945 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690091 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690228 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690363 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mmqc\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-kube-api-access-9mmqc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.690396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690388 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.690567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690402 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0217d3-06f8-425d-9978-1a3b7d817686-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:03.690736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690715 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690808 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.690962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.690940 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.692864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.692842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.692932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.692915 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.701919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.701892 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tzt\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.702032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.701968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:03.832074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:03.832036 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:04.021836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.021802 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:04.144773 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:11:04.144741 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5628ebf8_09d6_4d65_8346_093194994d8b.slice/crio-6d303dbee89c76171479b7a57cd1281bf7534e733e941102eeba4d0e3998b7b2 WatchSource:0}: Error finding container 6d303dbee89c76171479b7a57cd1281bf7534e733e941102eeba4d0e3998b7b2: Status 404 returned error can't find the container with id 6d303dbee89c76171479b7a57cd1281bf7534e733e941102eeba4d0e3998b7b2 Nov 01 22:11:04.146344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.146242 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:11:04.149026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.148553 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:11:04.149026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.148639 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:11:04.149026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.148685 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:11:04.431878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.431582 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" Nov 01 22:11:04.449372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.449345 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc" event={"ID":"6f0217d3-06f8-425d-9978-1a3b7d817686","Type":"ContainerDied","Data":"7eef58cbbcecca0fd2bb072dd27cac334660c3cfee6958830416b0cfe6409d85"} Nov 01 22:11:04.449751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.449376 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" event={"ID":"5628ebf8-09d6-4d65-8346-093194994d8b","Type":"ContainerStarted","Data":"3d9e0b4c16a46e3973a2408630d9c1a70d91987c8a8930eef5f45ba86210c7e7"} Nov 01 22:11:04.449751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.449401 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" event={"ID":"5628ebf8-09d6-4d65-8346-093194994d8b","Type":"ContainerStarted","Data":"6d303dbee89c76171479b7a57cd1281bf7534e733e941102eeba4d0e3998b7b2"} Nov 01 22:11:04.449751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.449430 5036 scope.go:122] "RemoveContainer" containerID="18f1eed6d67cbc7705b015cbc7f6bd9e2506993f0f41d0c8537a50704306fdef" Nov 01 22:11:04.475634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.475606 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:11:04.481374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.481345 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e2a91d-openshift-default-66f9fb6fc9-n9swc"] Nov 01 22:11:04.502768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.502727 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podStartSLOduration=38.502713313 podStartE2EDuration="38.502713313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:11:04.497402816 +0000 UTC m=+1618.190383296" watchObservedRunningTime="2026-11-01 22:11:04.502713313 +0000 UTC m=+1618.195693799" Nov 01 22:11:04.832698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.832663 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:04.913558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:04.913513 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:05.635045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:05.635004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:05.834646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:05.834613 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:05.834822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:05.834682 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:06.381471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:06.380931 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:06.516756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:06.516715 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" path="/var/lib/kubelet/pods/6f0217d3-06f8-425d-9978-1a3b7d817686/volumes" Nov 01 22:11:06.833209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:06.833167 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:06.833735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:06.833226 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:07.033554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:07.033420 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:07.824623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:07.824585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:07.841469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:07.832780 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:07.841469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:07.832848 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:08.641406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:08.641369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:08.835852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:08.835809 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:08.836025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:08.835882 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:09.505490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:09.504547 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:09.553714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:09.549324 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:09.563209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:09.563181 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:09.833253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:09.833213 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:09.833469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:09.833305 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:10.331486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:10.331402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:10.555660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:10.555626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:10.833210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:10.833175 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:11:10.833459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:10.833235 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 01 22:11:11.135238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:11.135153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:11.838127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:11.838090 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:11.838596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:11.838568 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:11.839423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:11.839403 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:11:11.844121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:11.844090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:12.635688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:12.635625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:13.353530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:13.353489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:14.090591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.090548 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:14.826637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.826578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:14.887833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.887798 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:11:14.890356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.890327 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0217d3-06f8-425d-9978-1a3b7d817686" containerName="istio-proxy" Nov 01 22:11:14.897221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.897188 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:14.921641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.921610 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3e300c\"/\"openshift-service-ca.crt\"" Nov 01 22:11:14.922508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.922434 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3e300c\"/\"default-dockercfg-m4xkm\"" Nov 01 22:11:14.922784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.922730 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3e300c\"/\"test-model-3e300c-kserve-self-signed-certs\"" Nov 01 22:11:14.922784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.922742 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3e300c\"/\"kube-root-ca.crt\"" Nov 01 22:11:14.934118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.934092 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:11:14.967313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:14.967111 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:15.002868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.002835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.003042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.002891 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.003271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.003244 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.003374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.003351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.003475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.003398 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.003475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.003433 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5cv\" (UniqueName: \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.106036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.105940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.106868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.106842 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.111554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.107386 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.112096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.112074 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5cv\" (UniqueName: \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.138465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.138356 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.138465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.138414 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.138807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.138779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.138990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.131327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5cv\" (UniqueName: \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.139092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.114918 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.139185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.107322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.139273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.106681 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.153175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.153147 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\") pod \"test-model-3e300c-kserve-7c8854b4b7-sn7tf\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.278929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.278677 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:15.357588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.354378 5036 scope.go:122] "RemoveContainer" containerID="aecb916e12a9b0adc3fe221582b52163bc667297928d73d6be3d7aa306c57c54" Nov 01 22:11:15.481830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.481806 5036 scope.go:122] "RemoveContainer" containerID="72f07ccc1b0c09425c857f509f8152e244e35faa91d0e46984566f1d826a54ab" Nov 01 22:11:15.525475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.525429 5036 scope.go:122] "RemoveContainer" containerID="dedda617f8a3be6303d53980e7dadfd744ff3e3122794ffb4689031babdf0cdf" Nov 01 22:11:15.559974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.559608 5036 scope.go:122] "RemoveContainer" containerID="399708a88c5b3d48d26081eed1603290cb3657de76d349fd07858e5a54885b35" Nov 01 22:11:15.689210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.689184 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:15.828478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:15.822497 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:11:16.344652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:16.344612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:16.661221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:16.661140 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" event={"ID":"88166992-e75c-43a0-86a5-49e3eec8ed60","Type":"ContainerStarted","Data":"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171"} Nov 01 22:11:16.661221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:16.661188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" event={"ID":"88166992-e75c-43a0-86a5-49e3eec8ed60","Type":"ContainerStarted","Data":"52070f59a1d1b8dd558ce06028036c4c6682079ec2f8f9a0a989422bb5c11eb9"} Nov 01 22:11:16.687113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:16.687054 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podStartSLOduration=2.687036831 podStartE2EDuration="2.687036831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:11:16.682913776 +0000 UTC m=+1630.375894265" watchObservedRunningTime="2026-11-01 22:11:16.687036831 +0000 UTC m=+1630.380017319" Nov 01 22:11:17.125521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:17.125430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:17.284966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:17.284925 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:17.797870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:17.797835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:18.624674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:18.624640 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:18.860400 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:11:18.860366 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:11:19.316467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:19.316421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:19.964315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:19.964277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:20.734553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:20.734508 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:21.426024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:21.425986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:22.090358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:22.090306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:22.747039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:22.747000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:22.993253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:22.993218 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:23.124169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:23.124133 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:23.487536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:23.487421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:24.254861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:24.254817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:24.934614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:24.934574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:25.284911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:25.284875 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:25.297963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:25.297926 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:25.628508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:25.628397 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:25.773841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:25.773795 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:11:26.505308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:26.505274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:27.352835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:27.352788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:28.038959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:28.038923 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:28.678009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:28.677969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:29.451259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:29.451220 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:30.137041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:30.137000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:30.765294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:30.765240 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:31.571431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:31.571391 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:32.431332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:32.431291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:33.081740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:33.081706 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:33.754350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:33.754312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:34.524910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:34.524855 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:35.226523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:35.226424 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:35.895241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:35.895201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:36.641660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:36.641623 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:37.336243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:37.336199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:37.976274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:37.976241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:38.613861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:38.613820 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:39.331898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:39.331851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:40.066149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:40.066108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:40.708194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:40.708147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:41.550700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:41.550664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:42.337235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:42.337202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:42.578591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:42.578553 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:11:42.578800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:42.578775 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" containerID="cri-o://5386de844a8fb690ad35676f2cc8af3c9d9dad1b22f9a96c07c23c5e625c8ef7" gracePeriod=30 Nov 01 22:11:43.032856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:43.032819 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:43.695593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:43.695553 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:44.427013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:44.426974 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:45.100527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:45.100477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:45.763207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:45.763169 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:46.537144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:46.537102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:47.236699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:47.236639 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:47.949122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:47.949036 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:48.123205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.123176 5036 generic.go:356] "Generic (PLEG): container finished" podID="050cdada-3493-47ec-9836-3d51ac6373e0" containerID="5386de844a8fb690ad35676f2cc8af3c9d9dad1b22f9a96c07c23c5e625c8ef7" exitCode=0 Nov 01 22:11:48.123388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.123261 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" event={"ID":"050cdada-3493-47ec-9836-3d51ac6373e0","Type":"ContainerDied","Data":"5386de844a8fb690ad35676f2cc8af3c9d9dad1b22f9a96c07c23c5e625c8ef7"} Nov 01 22:11:48.309650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.309622 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:11:48.348798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.348767 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:11:48.349629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.349610 5036 cpu_manager.go:414] "containerMap: removing container" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" Nov 01 22:11:48.349692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.349630 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" Nov 01 22:11:48.349845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.349834 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" containerName="istio-proxy" Nov 01 22:11:48.353916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.353889 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.361135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.359429 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-070e3d-serving-cert\"" Nov 01 22:11:48.361135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.359643 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:11:48.361135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.359711 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:11:48.361135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.360291 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:11:48.361135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.360391 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zltrj\"" Nov 01 22:11:48.365001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.364323 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457408 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457577 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457626 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457839 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.457957 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.458879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.458076 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.460140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.459826 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:11:48.460788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.460465 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:48.461358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.461333 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\") pod \"050cdada-3493-47ec-9836-3d51ac6373e0\" (UID: \"050cdada-3493-47ec-9836-3d51ac6373e0\") " Nov 01 22:11:48.461554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.461334 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:48.461638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.461578 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:48.461708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.461640 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:48.462225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462180 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7h5\" (UniqueName: \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.462575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.462853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462642 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.462853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462751 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.463007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462914 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.463007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462926 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050cdada-3493-47ec-9836-3d51ac6373e0-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.463007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462936 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.463007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462944 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.463007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.462953 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.463815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.463787 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:11:48.463899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.463808 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:11:48.463899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.463868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "kube-api-access-x5blz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:11:48.464937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.464919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo" pod "050cdada-3493-47ec-9836-3d51ac6373e0" (UID: "050cdada-3493-47ec-9836-3d51ac6373e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:11:48.565065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565029 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7h5\" (UniqueName: \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.565232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565159 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.565232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565203 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.565394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565250 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.565477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565407 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050cdada-3493-47ec-9836-3d51ac6373e0-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.565477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565430 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-kube-api-access-x5blz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.565477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565469 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050cdada-3493-47ec-9836-3d51ac6373e0-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.565635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565483 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050cdada-3493-47ec-9836-3d51ac6373e0-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:11:48.565903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.565871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.567738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.567714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.567821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.567746 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.577291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.577267 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7h5\" (UniqueName: \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\") pod \"maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.620859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.620827 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:48.678114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:48.677747 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:48.864698 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:11:48.864652 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:11:49.015177 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:11:49.015141 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93d83f2_2bcb_46ac_bb5e_6beee54eebb7.slice/crio-9df9eb4f3b347a660897ee64fc33a9e8dc1d25b210ad97cccb25ac3b3e0e2cac WatchSource:0}: Error finding container 9df9eb4f3b347a660897ee64fc33a9e8dc1d25b210ad97cccb25ac3b3e0e2cac: Status 404 returned error can't find the container with id 9df9eb4f3b347a660897ee64fc33a9e8dc1d25b210ad97cccb25ac3b3e0e2cac Nov 01 22:11:49.015663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.015290 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:11:49.017280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.017253 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:11:49.135078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.134986 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" event={"ID":"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7","Type":"ContainerStarted","Data":"9df9eb4f3b347a660897ee64fc33a9e8dc1d25b210ad97cccb25ac3b3e0e2cac"} Nov 01 22:11:49.136556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.136531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" Nov 01 22:11:49.136712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.136537 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8" event={"ID":"050cdada-3493-47ec-9836-3d51ac6373e0","Type":"ContainerDied","Data":"921d72ffaf3da920b81235335b9b693199c4ef8d2c99f70aecc8115ea177dd7c"} Nov 01 22:11:49.136712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.136656 5036 scope.go:122] "RemoveContainer" containerID="5386de844a8fb690ad35676f2cc8af3c9d9dad1b22f9a96c07c23c5e625c8ef7" Nov 01 22:11:49.167939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.167915 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:11:49.173278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.173251 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-2r6p8"] Nov 01 22:11:49.394755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:49.394611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:50.119373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:50.119281 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:50.448520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:50.448425 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050cdada-3493-47ec-9836-3d51ac6373e0" path="/var/lib/kubelet/pods/050cdada-3493-47ec-9836-3d51ac6373e0/volumes" Nov 01 22:11:50.896191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:50.896132 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:51.617106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:51.617063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:52.332388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:52.332348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:52.976155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:52.976122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:53.077735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:53.077699 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:11:53.277598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:53.277025 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" event={"ID":"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7","Type":"ContainerStarted","Data":"0e93ff3ed15dd768ab39669029f33a2d0c1765023f2f82cb73f4959b88449cf2"} Nov 01 22:11:53.301467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:53.282471 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:11:53.366466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:53.360608 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" podStartSLOduration=71.250941323 podStartE2EDuration="1m14.360588515s" totalImagesPullingTime="3.109647192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:11:53.31684938 +0000 UTC m=+1667.009829870" watchObservedRunningTime="2026-11-01 22:11:53.360588515 +0000 UTC m=+1667.053569001" Nov 01 22:11:53.958631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:53.958592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:55.423466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.423089 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:55.563469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.558673 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:11:55.667468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.666633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.689532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.687521 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.715468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.714527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.769471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.768542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.791471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.786523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.856471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.855680 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:55.943218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:55.942542 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.000470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.000091 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.159468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.158756 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:56.230527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.230340 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.254786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.251359 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.412480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.411530 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.416468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.413274 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.511537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.511054 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.518475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.517512 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.561466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.560531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.561466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.561124 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:56.809470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.808606 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.809470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.809170 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:11:56.964524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.956741 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:56.964524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:56.957272 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:11:57.098520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.087825 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": context deadline exceeded" Nov 01 22:11:57.102479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.101339 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.142011 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.142392 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.145283 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.232720 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.234538 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.235257 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:11:57.250513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.237581 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.256485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.256111 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:11:57.256916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.256117 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.257343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.257074 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:11:57.513030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.506275 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.513030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.506982 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:11:57.513030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.509704 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:11:57.513030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.509797 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:11:57.531569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.521858 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.531569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.530499 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:57.553509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.535757 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.553509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.536225 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.642518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.609981 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": context deadline exceeded" Nov 01 22:11:57.669662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.669623 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:11:57.669979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.669959 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:11:57.684470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.676728 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:11:57.729803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.686664 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.730392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.702388 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:11:57.730602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.703113 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.758618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.753972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:11:57.813318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.809027 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:57.813318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.809507 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:11:57.968077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:57.967009 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:58.145751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.145668 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:11:58.146434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.146404 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:58.147640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.147614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:58.148803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.148783 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:11:58.238156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.238113 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:11:58.238781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.238392 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:11:58.239298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.239277 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:11:58.239527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.239514 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:11:58.381602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.378647 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:11:58.413490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.413402 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:11:58.419462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.419420 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:11:58.421083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.421043 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:11:58.436379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.436359 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:11:58.472918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.472868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:58.499303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.499229 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:11:58.499528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.499468 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:11:58.499528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.499511 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:11:58.499696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:58.499542 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:11:59.320497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:59.320402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:11:59.458161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:11:59.457191 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:00.332498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:00.328201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:00.346201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:00.346173 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:12:01.021067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:01.021008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:02.119940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:02.119836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:02.259092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:02.259027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:02.957062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:02.957027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:03.707115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:03.707025 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:04.527317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:04.527275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:05.188467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:05.187774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:05.897468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:05.896837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:06.328361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:06.328313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:06.828205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:06.828124 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:08.119619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:08.119582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:08.901896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:08.901855 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:09.554241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:09.554193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:10.246705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:10.246666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:11.032024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:11.031990 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:11.940494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:11.940435 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:12.588931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:12.588898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:13.248322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.248284 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:13.347545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.347512 5036 generic.go:356] "Generic (PLEG): container finished" podID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerID="c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488" exitCode=0 Nov 01 22:12:13.347749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.347608 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerDied","Data":"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488"} Nov 01 22:12:13.348167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.348150 5036 scope.go:122] "RemoveContainer" containerID="c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488" Nov 01 22:12:13.350983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.350947 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:12:13.351152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.351036 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:12:13.351152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:13.351088 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:12:14.239239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:14.239207 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:14.361274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:14.361238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerStarted","Data":"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310"} Nov 01 22:12:15.040094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.040057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:15.324257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.324167 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:12:15.325779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.325753 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:15.325928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.325801 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:15.676915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.676825 5036 scope.go:122] "RemoveContainer" containerID="54a39a7b4a0e3e10b858c35a80ba2212ec99eb43dff9eebeb874f997ce768856" Nov 01 22:12:15.691579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.691553 5036 scope.go:122] "RemoveContainer" containerID="30f350bf8feac5210c748fe192f110875d4e81dd24dabe31f8eb6033ff610248" Nov 01 22:12:15.704988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.704969 5036 scope.go:122] "RemoveContainer" containerID="3c3ea4abfac61108393bd90bb287aaab1954f59975d4da668c6339af9bd305cd" Nov 01 22:12:15.756390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:15.756365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:16.324092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:16.324058 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:16.324331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:16.324124 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:16.541182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:16.541134 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:16.714307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:16.713965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:16.852868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:16.852835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:17.226885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:17.226824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:17.324064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:17.324023 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:17.324353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:17.324317 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:17.961924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:17.961882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:18.324710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:18.324670 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:18.325108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:18.324737 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:18.646950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:18.646872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:18.662602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:18.662523 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:12:18.861805 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:12:18.861768 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:12:19.324977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:19.324932 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:19.325544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:19.324995 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:19.350805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:19.350665 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:20.027030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:20.026992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:20.324597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:20.324513 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:20.324597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:20.324580 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:20.860145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:20.860108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:21.324617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:21.324583 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:21.324808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:21.324640 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:21.544252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:21.544214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:22.216369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:22.216334 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:22.325980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:22.325943 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:22.326216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:22.326012 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:23.165323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:23.164739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:23.327840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:23.327805 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:23.328262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:23.327863 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:23.909280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:23.909243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:24.324376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:24.324339 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Nov 01 22:12:24.324838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:24.324803 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Nov 01 22:12:24.682517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:24.682402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:25.330396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:25.330348 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:12:25.346532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:25.346498 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:12:25.423360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:25.423318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:26.742796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:26.742739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:27.683262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:27.683224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:28.463035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:28.462988 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:28.641510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:28.636785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:29.229850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:29.229813 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:30.090471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:30.087642 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:30.826345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:30.826306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:31.533417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:31.533383 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:32.353867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:32.353831 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:33.032995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:33.032946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:33.748235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:33.748203 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:34.546861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:34.546098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:35.277655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:35.277572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:36.014361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:36.014324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:36.676184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:36.676131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:38.124352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:38.124305 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:38.761963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:38.761927 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:39.691504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:39.691422 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:40.351307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:40.351268 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:41.052463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:41.052404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:41.822059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:41.822020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:42.654496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:42.654458 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:43.324721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:43.324681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:43.965818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:43.965754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:44.752018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:44.751984 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:44.959739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:44.959696 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:45.095856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:45.095817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:45.522456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:45.522404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:46.276661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:46.276623 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:47.139774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:47.139727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:47.962555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:47.962516 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:48.653699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:48.653661 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:48.862922 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:12:48.862887 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:12:49.328132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:49.328093 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:49.995541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:49.995501 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:50.685626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:50.685592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:51.513524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:51.513494 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:52.267178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:52.267147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:52.944197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:52.944155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:53.622293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:53.622252 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:54.320000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:54.319964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:55.054297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:55.054262 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:55.725583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:55.725534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:56.424462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:56.420958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:57.142134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:57.142003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:57.832236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:57.832197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:58.497185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:58.497135 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:59.248312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.248277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:59.452400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.452365 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:12:59.477226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.477171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:12:59.856025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.855995 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v"] Nov 01 22:12:59.860776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.860745 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:12:59.864518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.864494 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:12:59.864634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.864603 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:12:59.868236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:12:59.868210 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v"] Nov 01 22:13:00.008185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.008143 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.008376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.008317 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh524\" (UniqueName: \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.008506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.008484 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.110117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.110033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.110311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.110155 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh524\" (UniqueName: \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.110311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.110274 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.111004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.110979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.112760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.112734 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.122628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.122577 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh524\" (UniqueName: \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\") pod \"maas-api-revoke-sub-480e49cc-qz74v\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.147706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.141939 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:00.187569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.187530 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:00.512125 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:13:00.512088 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e16127_d4e9_4e13_a199_5d342357bd39.slice/crio-62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9 WatchSource:0}: Error finding container 62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9: Status 404 returned error can't find the container with id 62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9 Nov 01 22:13:00.512952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.512929 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v"] Nov 01 22:13:00.932455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:00.932398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:01.298511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:01.295749 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" event={"ID":"43e16127-d4e9-4e13-a199-5d342357bd39","Type":"ContainerStarted","Data":"62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9"} Nov 01 22:13:02.143153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:02.142749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:02.629163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:02.629111 5036 generic.go:356] "Generic (PLEG): container finished" podID="43e16127-d4e9-4e13-a199-5d342357bd39" containerID="db8401790f855e1c421ac7be6ec7070e02408e088284448ddcbdadf60f810693" exitCode=0 Nov 01 22:13:02.629502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:02.629475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" event={"ID":"43e16127-d4e9-4e13-a199-5d342357bd39","Type":"ContainerDied","Data":"db8401790f855e1c421ac7be6ec7070e02408e088284448ddcbdadf60f810693"} Nov 01 22:13:03.094629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:03.094585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:03.835936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:03.835872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:03.905862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:03.905789 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:04.006147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.006102 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\") pod \"43e16127-d4e9-4e13-a199-5d342357bd39\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " Nov 01 22:13:04.006353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.006284 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\" (UniqueName: \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\") pod \"43e16127-d4e9-4e13-a199-5d342357bd39\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " Nov 01 22:13:04.006421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.006409 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\") pod \"43e16127-d4e9-4e13-a199-5d342357bd39\" (UID: \"43e16127-d4e9-4e13-a199-5d342357bd39\") " Nov 01 22:13:04.008259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.008227 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca" pod "43e16127-d4e9-4e13-a199-5d342357bd39" (UID: "43e16127-d4e9-4e13-a199-5d342357bd39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:04.019760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.014208 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524" pod "43e16127-d4e9-4e13-a199-5d342357bd39" (UID: "43e16127-d4e9-4e13-a199-5d342357bd39"). InnerVolumeSpecName "kube-api-access-qh524". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:04.019760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.015194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca" pod "43e16127-d4e9-4e13-a199-5d342357bd39" (UID: "43e16127-d4e9-4e13-a199-5d342357bd39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:04.110185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.110150 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:04.110185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.110180 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e16127-d4e9-4e13-a199-5d342357bd39-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:04.110185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.110191 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh524\" (UniqueName: \"kubernetes.io/projected/43e16127-d4e9-4e13-a199-5d342357bd39-kube-api-access-qh524\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:04.648931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.648895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:04.650668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.650635 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" Nov 01 22:13:04.650829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.650665 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v" event={"ID":"43e16127-d4e9-4e13-a199-5d342357bd39","Type":"ContainerDied","Data":"62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9"} Nov 01 22:13:04.650829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:04.650713 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b615182d5e6c2c34a7a9ad7b266ab3ca0a47375a840fa426b973444bd2d0b9" Nov 01 22:13:05.501665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:05.501623 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:06.226530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:06.226495 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:06.897805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:06.897772 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:07.556427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:07.556387 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:08.219795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:08.219759 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:08.870669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:08.870636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:09.544383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:09.544303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:10.229428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:10.229393 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:10.534309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:10.534273 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:13:10.711468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:10.705111 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:11.077292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.077252 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t"] Nov 01 22:13:11.078654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.078629 5036 cpu_manager.go:414] "containerMap: removing container" podUID="43e16127-d4e9-4e13-a199-5d342357bd39" containerName="revoke-keys" Nov 01 22:13:11.078654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.078656 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e16127-d4e9-4e13-a199-5d342357bd39" containerName="revoke-keys" Nov 01 22:13:11.078914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.078901 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e16127-d4e9-4e13-a199-5d342357bd39" containerName="revoke-keys" Nov 01 22:13:11.083619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.083593 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.090264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.090237 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t"] Nov 01 22:13:11.115698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.115670 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:13:11.122463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.122408 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:13:11.192158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.192124 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.192333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.192309 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.192487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.192472 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbc9d\" (UniqueName: \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.294121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.294089 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.294289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.294228 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbc9d\" (UniqueName: \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.294429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.294413 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.295131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.295106 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.296678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.296653 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.306774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.306744 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbc9d\" (UniqueName: \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\") pod \"maas-api-revoke-sub-0c8b0dbc-6tl8t\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.424367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.424277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:11.430203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.430124 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:11.840199 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:13:11.840171 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8563d3c_b9b0_4946_ac7f_8abb2e26f1a7.slice/crio-143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee WatchSource:0}: Error finding container 143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee: Status 404 returned error can't find the container with id 143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee Nov 01 22:13:11.841500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:11.841406 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t"] Nov 01 22:13:12.110218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:12.110172 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:12.791878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:12.791842 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" event={"ID":"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7","Type":"ContainerStarted","Data":"9d3c9c47aae6ced0f52be758c96bb4708b9f89aa6ac9afd15f15a55d91aca87f"} Nov 01 22:13:12.792066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:12.791899 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" event={"ID":"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7","Type":"ContainerStarted","Data":"143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee"} Nov 01 22:13:12.821859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:12.821793 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" podStartSLOduration=1.821773906 podStartE2EDuration="1.821773906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:12.816147229 +0000 UTC m=+1746.509127716" watchObservedRunningTime="2026-11-01 22:13:12.821773906 +0000 UTC m=+1746.514754396" Nov 01 22:13:12.875171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:12.875133 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:13.667466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:13.664644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:13.804464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:13.804409 5036 generic.go:356] "Generic (PLEG): container finished" podID="b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" containerID="9d3c9c47aae6ced0f52be758c96bb4708b9f89aa6ac9afd15f15a55d91aca87f" exitCode=0 Nov 01 22:13:13.804673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:13.804478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" event={"ID":"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7","Type":"ContainerDied","Data":"9d3c9c47aae6ced0f52be758c96bb4708b9f89aa6ac9afd15f15a55d91aca87f"} Nov 01 22:13:14.341289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:14.341254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:14.469861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:14.469822 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:15.027924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.027889 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:15.131199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.131110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:15.133581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.133557 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\" (UniqueName: \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\") pod \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " Nov 01 22:13:15.133793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.133773 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\") pod \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " Nov 01 22:13:15.133879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.133814 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\") pod \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\" (UID: \"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7\") " Nov 01 22:13:15.135410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.134208 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca" pod "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" (UID: "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:15.135410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.134772 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:15.159514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.150837 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca" pod "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" (UID: "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:15.221464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.218308 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d" pod "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" (UID: "b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7"). InnerVolumeSpecName "kube-api-access-fbc9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:15.239215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.238987 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbc9d\" (UniqueName: \"kubernetes.io/projected/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-kube-api-access-fbc9d\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:15.239215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.239013 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:15.759227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.759188 5036 scope.go:122] "RemoveContainer" containerID="c5ff86b0bfafa9d14f367f35dc437878f06d88ea5ece787c2ed3c473e77ceaf6" Nov 01 22:13:15.829402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.829372 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" Nov 01 22:13:15.829566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.829405 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t" event={"ID":"b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7","Type":"ContainerDied","Data":"143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee"} Nov 01 22:13:15.829566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.829476 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143998bdd0108d127a3801d3455cf59e33a5f1f2ed73492d10472f88ea9e02ee" Nov 01 22:13:15.925182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:15.925143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:16.621875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:16.621810 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:17.395833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:17.395790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:18.140198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:18.140159 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:18.854396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:18.854362 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:18.933014 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:18.932975 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:13:19.638114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:19.638069 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:20.446493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:20.446435 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:20.566222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:20.566182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:21.217597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:21.217560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:21.945101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:21.945047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:22.745780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:22.745743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:23.267017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.266923 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:13:23.292503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.291648 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:23.671578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.671478 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd"] Nov 01 22:13:23.673748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.673717 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" containerName="revoke-keys" Nov 01 22:13:23.673900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.673769 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" containerName="revoke-keys" Nov 01 22:13:23.674220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.674201 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" containerName="revoke-keys" Nov 01 22:13:23.679999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.679969 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.680356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.680331 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd"] Nov 01 22:13:23.686976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.684589 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:13:23.686976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.684648 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:13:23.794017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.793981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.794366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.794295 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.794556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.794429 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f478f\" (UniqueName: \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.896732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.896696 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.896919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.896785 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.896919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.896834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f478f\" (UniqueName: \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.897565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.897514 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.899268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.899243 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.909026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.908993 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f478f\" (UniqueName: \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\") pod \"maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:23.925982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:23.925887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:24.024123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:24.024074 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:24.339015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:24.338987 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd"] Nov 01 22:13:24.698182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:24.698140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:24.990982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:24.990898 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" event={"ID":"a7d05ea0-a366-400d-a2bf-5d750e50c842","Type":"ContainerStarted","Data":"66d9ebb5b16f8b974cc95d377d6a3e4ee4e7d7aa74cd370e4e47633246cb8c78"} Nov 01 22:13:24.990982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:24.990955 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" event={"ID":"a7d05ea0-a366-400d-a2bf-5d750e50c842","Type":"ContainerStarted","Data":"ca0ce713e3233db7d951b1f43454408898c3c858063b8ca42a6549265038aeb0"} Nov 01 22:13:25.015302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.015250 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" podStartSLOduration=2.015236301 podStartE2EDuration="2.015236301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:25.008545055 +0000 UTC m=+1758.701525543" watchObservedRunningTime="2026-11-01 22:13:25.015236301 +0000 UTC m=+1758.708216787" Nov 01 22:13:25.410440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.410397 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:25.558178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.558092 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" containerID="cri-o://bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8" gracePeriod=45 Nov 01 22:13:25.894605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.894557 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:13:25.946156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.946156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.946348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946189 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.946348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946218 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.946428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946396 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:25.946635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946608 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:25.946635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946624 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\" (UniqueName: \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.946777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\") pod \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\" (UID: \"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1\") " Nov 01 22:13:25.947011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.946981 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:25.947410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.947392 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:25.947504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.947413 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:25.947504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.947423 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:25.948693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.948668 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:25.948798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.948778 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:25.948940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:25.948924 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64" pod "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" (UID: "4eb78bdd-d5e7-4503-9e44-9e69cbb447f1"). InnerVolumeSpecName "kube-api-access-6rw64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:26.006841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.006808 5036 generic.go:356] "Generic (PLEG): container finished" podID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerID="bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8" exitCode=0 Nov 01 22:13:26.007014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.006866 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" Nov 01 22:13:26.007014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.006895 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" event={"ID":"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1","Type":"ContainerDied","Data":"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8"} Nov 01 22:13:26.007014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.006949 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p" event={"ID":"4eb78bdd-d5e7-4503-9e44-9e69cbb447f1","Type":"ContainerDied","Data":"c5cccc317d9dc724666355524caee76097906d9fc279455d2810fe5c8671c96f"} Nov 01 22:13:26.007014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.006975 5036 scope.go:122] "RemoveContainer" containerID="bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8" Nov 01 22:13:26.009244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.009222 5036 generic.go:356] "Generic (PLEG): container finished" podID="a7d05ea0-a366-400d-a2bf-5d750e50c842" containerID="66d9ebb5b16f8b974cc95d377d6a3e4ee4e7d7aa74cd370e4e47633246cb8c78" exitCode=0 Nov 01 22:13:26.009423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.009404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" event={"ID":"a7d05ea0-a366-400d-a2bf-5d750e50c842","Type":"ContainerDied","Data":"66d9ebb5b16f8b974cc95d377d6a3e4ee4e7d7aa74cd370e4e47633246cb8c78"} Nov 01 22:13:26.021922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.021899 5036 scope.go:122] "RemoveContainer" containerID="bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8" Nov 01 22:13:26.022217 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:26.022194 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8\": container with ID starting with bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8 not found: ID does not exist" containerID="bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8" Nov 01 22:13:26.022312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.022222 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8"} err="failed to get container status \"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8\": rpc error: code = NotFound desc = could not find container \"bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8\": container with ID starting with bd0db61446cbad27dd86a949edb4f891d98fd3d68e4dc1684f63d9af654548e8 not found: ID does not exist" Nov 01 22:13:26.049363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.049334 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rw64\" (UniqueName: \"kubernetes.io/projected/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-kube-api-access-6rw64\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:26.049552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.049370 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:26.049552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.049386 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:26.067822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.067799 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:13:26.075312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.075286 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-edd32a/test-model-edd32a-kserve-75ff95445-jbq4p"] Nov 01 22:13:26.310829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.310798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:26.439538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.439488 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:26.449311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:26.449275 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" path="/var/lib/kubelet/pods/4eb78bdd-d5e7-4503-9e44-9e69cbb447f1/volumes" Nov 01 22:13:27.184765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.164743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:27.327620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.327596 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:27.398256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.398220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\" (UniqueName: \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\") pod \"a7d05ea0-a366-400d-a2bf-5d750e50c842\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " Nov 01 22:13:27.398509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.398417 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\") pod \"a7d05ea0-a366-400d-a2bf-5d750e50c842\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " Nov 01 22:13:27.398717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.398691 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\") pod \"a7d05ea0-a366-400d-a2bf-5d750e50c842\" (UID: \"a7d05ea0-a366-400d-a2bf-5d750e50c842\") " Nov 01 22:13:27.415739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.401101 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca" pod "a7d05ea0-a366-400d-a2bf-5d750e50c842" (UID: "a7d05ea0-a366-400d-a2bf-5d750e50c842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:27.415739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.402625 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca" pod "a7d05ea0-a366-400d-a2bf-5d750e50c842" (UID: "a7d05ea0-a366-400d-a2bf-5d750e50c842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:27.416148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.416111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f" pod "a7d05ea0-a366-400d-a2bf-5d750e50c842" (UID: "a7d05ea0-a366-400d-a2bf-5d750e50c842"). InnerVolumeSpecName "kube-api-access-f478f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:27.501843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.501795 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f478f\" (UniqueName: \"kubernetes.io/projected/a7d05ea0-a366-400d-a2bf-5d750e50c842-kube-api-access-f478f\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:27.501843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.501829 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:27.501843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.501845 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d05ea0-a366-400d-a2bf-5d750e50c842-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:27.864108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:27.864071 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:28.034068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:28.034037 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" Nov 01 22:13:28.034068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:28.034058 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd" event={"ID":"a7d05ea0-a366-400d-a2bf-5d750e50c842","Type":"ContainerDied","Data":"ca0ce713e3233db7d951b1f43454408898c3c858063b8ca42a6549265038aeb0"} Nov 01 22:13:28.034279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:28.034130 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0ce713e3233db7d951b1f43454408898c3c858063b8ca42a6549265038aeb0" Nov 01 22:13:28.642370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:28.642331 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:29.320878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:29.320839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:30.023047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:30.022986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:30.750408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:30.750363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:31.459470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:31.450881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:32.166832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:32.166798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:32.921232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:32.921194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:33.566195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:33.566153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:33.964474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:33.964418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:34.126640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:34.123710 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:34.758551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:34.758507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:35.447568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:35.447530 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:36.240758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:36.240704 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:36.915123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:36.915084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:37.649089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:37.649054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:38.297438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.297393 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" containerID="cri-o://8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13" gracePeriod=45 Nov 01 22:13:38.433324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.433285 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": dial tcp 10.132.0.192:8000: connect: connection refused" Nov 01 22:13:38.435735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.435709 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:38.720956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.720925 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:13:38.823098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823008 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\" (UniqueName: \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.823291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823215 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.823360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823314 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.823420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.823639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823621 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.823719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.823700 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\") pod \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\" (UID: \"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4\") " Nov 01 22:13:38.824962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.824930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:38.825177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.825154 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:38.825428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.825401 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:38.826132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.826111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:38.826338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.826321 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "kube-api-access-bv8fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:38.826835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.826812 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm" pod "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" (UID: "ea2543f8-15fe-47b2-a4a9-a2f4b783dba4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:38.927366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927335 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8fs\" (UniqueName: \"kubernetes.io/projected/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-kube-api-access-bv8fs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:38.927366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927366 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:38.927366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927376 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:38.927602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927387 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:38.927602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927396 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:38.927602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:38.927404 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:39.218526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.194358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:39.332140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.332098 5036 generic.go:356] "Generic (PLEG): container finished" podID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerID="8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13" exitCode=0 Nov 01 22:13:39.332623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.332162 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" Nov 01 22:13:39.332623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.332159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" event={"ID":"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4","Type":"ContainerDied","Data":"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13"} Nov 01 22:13:39.332623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.332298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k" event={"ID":"ea2543f8-15fe-47b2-a4a9-a2f4b783dba4","Type":"ContainerDied","Data":"d647e3bfc970d649c83347d9b0a05dd4186467d50c9a86d01328900fd5b43a83"} Nov 01 22:13:39.332623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.332336 5036 scope.go:122] "RemoveContainer" containerID="8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13" Nov 01 22:13:39.345546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.345522 5036 scope.go:122] "RemoveContainer" containerID="8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13" Nov 01 22:13:39.345809 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:39.345791 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13\": container with ID starting with 8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13 not found: ID does not exist" containerID="8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13" Nov 01 22:13:39.345859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.345816 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13"} err="failed to get container status \"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13\": rpc error: code = NotFound desc = could not find container \"8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13\": container with ID starting with 8f596b2e3a9d0224b5dd74a457689df6c524b396e4a60edaf3ce23cb21fcbe13 not found: ID does not exist" Nov 01 22:13:39.359588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.359562 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:13:39.365781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.365758 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3577a0/test-model-3577a0-kserve-789f498ff7-m7g2k"] Nov 01 22:13:39.942644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:39.942607 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:40.442044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:40.442015 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" path="/var/lib/kubelet/pods/ea2543f8-15fe-47b2-a4a9-a2f4b783dba4/volumes" Nov 01 22:13:40.830626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:40.830595 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:41.516841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:41.516802 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:42.545267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:42.545226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:43.402019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:43.401977 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:44.226121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:44.226083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:44.951137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:44.951101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:45.031120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:45.031080 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm"] Nov 01 22:13:45.038573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:45.037782 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745e52b1-ltqmm"] Nov 01 22:13:45.640220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:45.640181 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:46.327859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:46.327818 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:46.449832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:46.449796 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4feac34-c305-47e9-81d8-c36df6d71c20" path="/var/lib/kubelet/pods/f4feac34-c305-47e9-81d8-c36df6d71c20/volumes" Nov 01 22:13:47.142030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.141988 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:47.535480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.534598 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:13:47.535480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.534900 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" containerID="cri-o://c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff" gracePeriod=30 Nov 01 22:13:47.542957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.542925 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:13:47.543149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.543126 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" containerID="cri-o://63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677" gracePeriod=30 Nov 01 22:13:47.548636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.548609 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:13:47.548836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.548814 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" containerID="cri-o://299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c" gracePeriod=30 Nov 01 22:13:47.749623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.749526 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:47.914195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:47.912245 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:48.062762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:48.062725 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:48.537044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:48.536594 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd"] Nov 01 22:13:48.539201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:48.539164 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edd32a-h6xvd"] Nov 01 22:13:48.540878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:48.540851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:48.754594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:48.754393 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:48.863985 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:48.863894 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:13:49.454686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:49.454649 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:49.501539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:49.498281 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 22:13:49.501539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:49.498535 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" containerID="cri-o://cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b" gracePeriod=30 Nov 01 22:13:49.956338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:49.956305 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 22:13:49.956800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:49.956510 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" containerID="cri-o://c32b88e92c630ccf9335c658e7f2308464fde0830b6142be5646ae90b5b89a58" gracePeriod=30 Nov 01 22:13:50.338041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:50.337995 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:50.456147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:50.456111 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d05ea0-a366-400d-a2bf-5d750e50c842" path="/var/lib/kubelet/pods/a7d05ea0-a366-400d-a2bf-5d750e50c842/volumes" Nov 01 22:13:51.137371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:51.137326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:51.849883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:51.849840 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:52.746874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:52.745972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:52.984504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:52.982846 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.032136 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033391 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a7d05ea0-a366-400d-a2bf-5d750e50c842" containerName="revoke-keys" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033414 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d05ea0-a366-400d-a2bf-5d750e50c842" containerName="revoke-keys" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033465 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033473 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033556 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033564 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033574 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033581 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033785 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerName="istio-proxy" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033814 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2543f8-15fe-47b2-a4a9-a2f4b783dba4" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033822 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb78bdd-d5e7-4503-9e44-9e69cbb447f1" containerName="main" Nov 01 22:13:53.037318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.033834 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d05ea0-a366-400d-a2bf-5d750e50c842" containerName="revoke-keys" Nov 01 22:13:53.049250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.038430 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.049250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.043967 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3e300c-openshift-default-dockercfg-lpvl5\"" Nov 01 22:13:53.056753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.056728 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:13:53.097361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.097332 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:13:53.114461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.111608 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150104 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150171 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150205 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150304 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150339 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150372 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150406 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150462 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150514 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150598 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150711 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150792 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.150972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.151042 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154460 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154497 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154556 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154638 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154746 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\") pod \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\" (UID: \"ba92ad64-e063-4f45-acb7-7258c8dfc1e0\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.154806 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155469 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7tz\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155535 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155629 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155730 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.155792 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.156014 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.156081 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.156401 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.156878 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.157173 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.157341 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.158815 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.158998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.159271 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.159330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.159595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.159711 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160064 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160332 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.160533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160611 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.160624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "kube-api-access-ls2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.162471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "kube-api-access-qmng2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.162800 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.163042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.163337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.163084 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "kube-api-access-mzgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.164186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.164150 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:53.164785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.164762 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.166357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.166330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.167969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.167950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:53.183507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.174335 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.184918 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195509 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195557 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195579 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195611 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195616 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195789 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195802 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerName="istio-proxy" Nov 01 22:13:53.196364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.195813 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerName="istio-proxy" Nov 01 22:13:53.198198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.190132 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo" pod "ba92ad64-e063-4f45-acb7-7258c8dfc1e0" (UID: "ba92ad64-e063-4f45-acb7-7258c8dfc1e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:53.258166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.257988 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.272464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.259641 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:13:53.272464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.259711 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:13:53.272464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.264136 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.277238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.277203 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.277768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.277745 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:13:53.278079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.278009 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.278604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.278583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") pod \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\" (UID: \"9c1fb377-5ded-45dc-a75f-f98b4a3caf70\") " Nov 01 22:13:53.278790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.278772 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") pod \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\" (UID: \"5ee2ae42-241a-44a2-9332-72502fd4c5ff\") " Nov 01 22:13:53.279715 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:13:53.279699 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/9c1fb377-5ded-45dc-a75f-f98b4a3caf70/volumes/kubernetes.io~projected/istio-token Nov 01 22:13:53.279912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.279886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token" pod "9c1fb377-5ded-45dc-a75f-f98b4a3caf70" (UID: "9c1fb377-5ded-45dc-a75f-f98b4a3caf70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.280332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.280306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.281108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.281088 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.281334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.281314 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7tz\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.281528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.281502 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.283627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.280945 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.284933 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:13:53.280514 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/5ee2ae42-241a-44a2-9332-72502fd4c5ff/volumes/kubernetes.io~projected/istio-token Nov 01 22:13:53.284933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.284601 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token" pod "5ee2ae42-241a-44a2-9332-72502fd4c5ff" (UID: "5ee2ae42-241a-44a2-9332-72502fd4c5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:53.284933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.283821 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.284933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.284719 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.284933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.284791 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.285721 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.285805 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286110 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286140 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286156 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286171 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286185 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286205 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286225 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls2bz\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-kube-api-access-ls2bz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286240 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286256 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286277 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286292 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286312 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286334 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286350 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzgkr\" (UniqueName: \"kubernetes.io/projected/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-kube-api-access-mzgkr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286366 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286382 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286402 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286418 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286434 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286532 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286554 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286573 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286589 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286603 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286625 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2ae42-241a-44a2-9332-72502fd4c5ff-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286643 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmng2\" (UniqueName: \"kubernetes.io/projected/ba92ad64-e063-4f45-acb7-7258c8dfc1e0-kube-api-access-qmng2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.286663 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1fb377-5ded-45dc-a75f-f98b4a3caf70-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.282379 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.288410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.288086 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.291087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.289343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.292103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.291988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.327009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.326871 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7tz\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.330278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.329054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.330962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.330939 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.334710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.334670 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.388293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388261 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.388469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388313 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388485 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.388644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388628 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.388687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388672 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388808 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hz7\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.388963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388923 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbqz\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.388963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.388953 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.389100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.389139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389112 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.389182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389151 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.389224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389200 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.389266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389250 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.389341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389328 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.389378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.389351 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.393074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.393050 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:53.495070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.494801 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.494874 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.494905 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.494968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495141 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495179 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495278 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495419 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495480 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hz7\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495515 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495542 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbqz\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495582 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495740 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495776 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.495993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495897 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.496664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.496833 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.497176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.495511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.497439 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.498104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.498176 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.498495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.498770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.498728 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.499360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.499001 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.501191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.501163 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.501829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.501803 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.503999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.503972 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.518400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.518355 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.518877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.518841 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.519404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.519365 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbqz\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.521387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.521113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hz7\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.541320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.539994 5036 generic.go:356] "Generic (PLEG): container finished" podID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" containerID="c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff" exitCode=0 Nov 01 22:13:53.541320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.540257 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" Nov 01 22:13:53.541320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.540799 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" event={"ID":"5ee2ae42-241a-44a2-9332-72502fd4c5ff","Type":"ContainerDied","Data":"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff"} Nov 01 22:13:53.541320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.540860 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5" event={"ID":"5ee2ae42-241a-44a2-9332-72502fd4c5ff","Type":"ContainerDied","Data":"89a9cdb37d2264eb8ec267e5b9a78c754605fb86cecb0d64d8fa2a5dcfbe55b2"} Nov 01 22:13:53.541320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.540889 5036 scope.go:122] "RemoveContainer" containerID="c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff" Nov 01 22:13:53.543739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.543123 5036 generic.go:356] "Generic (PLEG): container finished" podID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" containerID="63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677" exitCode=0 Nov 01 22:13:53.543739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.543295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" event={"ID":"ba92ad64-e063-4f45-acb7-7258c8dfc1e0","Type":"ContainerDied","Data":"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677"} Nov 01 22:13:53.543739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.543338 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" event={"ID":"ba92ad64-e063-4f45-acb7-7258c8dfc1e0","Type":"ContainerDied","Data":"cdcd53c8c0c4d04b4eb1c39fb861e8a7621538daf57ba8beb95a6c8d92c233a2"} Nov 01 22:13:53.543739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.543423 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp" Nov 01 22:13:53.551780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.551620 5036 generic.go:356] "Generic (PLEG): container finished" podID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" containerID="299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c" exitCode=0 Nov 01 22:13:53.552172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.551807 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerDied","Data":"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c"} Nov 01 22:13:53.552172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.551856 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" event={"ID":"9c1fb377-5ded-45dc-a75f-f98b4a3caf70","Type":"ContainerDied","Data":"14ae4d926db23ad24a60a84159f33c24ad8a2eb1c13d5de97dde8baff8ce2e15"} Nov 01 22:13:53.552172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.551953 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq" Nov 01 22:13:53.561332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.561309 5036 scope.go:122] "RemoveContainer" containerID="c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff" Nov 01 22:13:53.563438 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:53.563414 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff\": container with ID starting with c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff not found: ID does not exist" containerID="c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff" Nov 01 22:13:53.563655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.563614 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff"} err="failed to get container status \"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff\": rpc error: code = NotFound desc = could not find container \"c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff\": container with ID starting with c647c2250c980777dcb47bfdfbd60ad398eeee2b415351074aa553ebebfd8cff not found: ID does not exist" Nov 01 22:13:53.563655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.563647 5036 scope.go:122] "RemoveContainer" containerID="63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677" Nov 01 22:13:53.622605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.622579 5036 scope.go:122] "RemoveContainer" containerID="63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677" Nov 01 22:13:53.623085 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:53.623064 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677\": container with ID starting with 63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677 not found: ID does not exist" containerID="63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677" Nov 01 22:13:53.623176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.623092 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677"} err="failed to get container status \"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677\": rpc error: code = NotFound desc = could not find container \"63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677\": container with ID starting with 63ae65249acfeaee13ba6fa0c293c518886c86cd2125b1f09a4bd02c91a19677 not found: ID does not exist" Nov 01 22:13:53.623176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.623113 5036 scope.go:122] "RemoveContainer" containerID="299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c" Nov 01 22:13:53.624329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.624314 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:53.633381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.633360 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:13:53.644186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.643652 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-hmkt5"] Nov 01 22:13:53.644882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.644859 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:53.645507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.645487 5036 scope.go:122] "RemoveContainer" containerID="85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2" Nov 01 22:13:53.664993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.662655 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:13:53.672280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.672255 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mkhdp"] Nov 01 22:13:53.690952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.690896 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:13:53.725662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.725272 5036 scope.go:122] "RemoveContainer" containerID="299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c" Nov 01 22:13:53.726199 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:53.726023 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c\": container with ID starting with 299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c not found: ID does not exist" containerID="299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c" Nov 01 22:13:53.726199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.726067 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c"} err="failed to get container status \"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c\": rpc error: code = NotFound desc = could not find container \"299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c\": container with ID starting with 299b5bf9000aefb4bfaee8d62291bd0e0d48b36a0b7c984b82b0cf048f74dc8c not found: ID does not exist" Nov 01 22:13:53.726199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.726095 5036 scope.go:122] "RemoveContainer" containerID="85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2" Nov 01 22:13:53.731680 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:53.729689 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2\": container with ID starting with 85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2 not found: ID does not exist" containerID="85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2" Nov 01 22:13:53.731680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.729722 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2"} err="failed to get container status \"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2\": rpc error: code = NotFound desc = could not find container \"85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2\": container with ID starting with 85a4ae08652bce4d338bff9ba6f44582478500aa75e640c01f541006f75a64e2 not found: ID does not exist" Nov 01 22:13:53.766781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.759889 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-nkqsq"] Nov 01 22:13:53.786619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.786145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:53.812283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.788729 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:53.812283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.788822 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:53.812283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.788867 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:53.872850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:53.846832 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:13:54.141424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.138271 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:13:54.214362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.193641 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.214362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.193739 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.214362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.193804 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.253076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.245926 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.253076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.246025 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.253076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.246069 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:13:54.398562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.394277 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:13:54.494424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.494372 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee2ae42-241a-44a2-9332-72502fd4c5ff" path="/var/lib/kubelet/pods/5ee2ae42-241a-44a2-9332-72502fd4c5ff/volumes" Nov 01 22:13:54.503893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.503864 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1fb377-5ded-45dc-a75f-f98b4a3caf70" path="/var/lib/kubelet/pods/9c1fb377-5ded-45dc-a75f-f98b4a3caf70/volumes" Nov 01 22:13:54.504852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.504833 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba92ad64-e063-4f45-acb7-7258c8dfc1e0" path="/var/lib/kubelet/pods/ba92ad64-e063-4f45-acb7-7258c8dfc1e0/volumes" Nov 01 22:13:54.575553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.575517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" event={"ID":"12da8b2d-cbec-446a-957a-2bd7eae29773","Type":"ContainerStarted","Data":"95d09e90199fe0225f40d01e7c954b2d71f7123577a977fbc45905b877c74756"} Nov 01 22:13:54.589327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.589242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" event={"ID":"52e28a5c-aaaa-4698-9c75-62d149c5c19b","Type":"ContainerStarted","Data":"bf2e7730236f7e8afcf96aa590ab2513eb48e203067524cb2b9f189c9784b853"} Nov 01 22:13:54.593699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.593416 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" event={"ID":"6fb8b132-d7f0-4db8-8fc4-34b424107dd2","Type":"ContainerStarted","Data":"2e0ef8af766fba809f39684f635afbba27b6c462dc661126366a9034fc510156"} Nov 01 22:13:54.593699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.593489 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" event={"ID":"6fb8b132-d7f0-4db8-8fc4-34b424107dd2","Type":"ContainerStarted","Data":"14bd745e8495f76e53a582f4cf7f390a8201717d905b6e31f7d46cd9e78d831f"} Nov 01 22:13:54.597836 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:13:54.597806 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f88cf_9031_4d5e_86fe_233ae699a781.slice/crio-conmon-cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:13:54.631173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:54.631130 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podStartSLOduration=98.631115347 podStartE2EDuration="1m38.631115347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:54.627557816 +0000 UTC m=+1788.320538302" watchObservedRunningTime="2026-11-01 22:13:54.631115347 +0000 UTC m=+1788.324095833" Nov 01 22:13:55.662621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:55.489887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:13:55.662621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:55.490501 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:13:55.662621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:55.641480 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:55.915044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:55.790966 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:13:56.320526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.294233 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/health\": context deadline exceeded" Nov 01 22:13:56.524504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.444766 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.524504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.446192 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:13:56.524504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.446257 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:13:56.524504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.519487 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.610295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.590033 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.630418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.612424 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:13:56.634624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.612503 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:13:56.634814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.614601 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.634941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.614614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:13:56.822478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.820521 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.866109 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.867263 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.892531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.893587 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.894625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.895650 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:13:56.917566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:56.896702 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:13:57.038472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.030758 5036 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 22:13:57.038794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.038758 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 22:13:57.134843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.134783 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:13:57.171178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.171089 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.171469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.171414 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.171765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.171747 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.172061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.172044 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.172248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.172233 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.207012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.206983 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.207337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.207316 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.207528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.207510 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.207686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.207670 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:57.250521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.211645 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:57.250848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.250806 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:57.314270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.314211 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:57.334469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.315305 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:57.335419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.316338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:57.337363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.337327 5036 generic.go:356] "Generic (PLEG): container finished" podID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerID="cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b" exitCode=0 Nov 01 22:13:57.337771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.337724 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" event={"ID":"a39f88cf-9031-4d5e-86fe-233ae699a781","Type":"ContainerDied","Data":"cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b"} Nov 01 22:13:57.338054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.338035 5036 scope.go:122] "RemoveContainer" containerID="cc8b2b8a75d05d8c8f7e0fa345a9e38a20e785523940e3a7d78b00881c5eee1b" Nov 01 22:13:57.354537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.354508 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" Nov 01 22:13:57.461680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.423179 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:13:57.461680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.433190 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:13:57.461680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.452561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.487545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.487497 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:57.687270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.687212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751387 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751425 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751457 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751472 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751486 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f88cf-9031-4d5e-86fe-233ae699a781-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.752541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.751499 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f88cf-9031-4d5e-86fe-233ae699a781-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.795931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.777581 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:13:57.795931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.777646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:13:57.795931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.779697 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.797034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.796353 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.808988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.807906 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.808988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.808203 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/health\": context deadline exceeded" Nov 01 22:13:57.815471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.810853 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:13:57.815471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.810919 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:13:57.815471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.810959 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:13:57.815471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.811018 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:13:57.872944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.869833 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.961063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.873509 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.962323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.875241 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.875697 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.876480 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.906957 5036 generic.go:356] "Generic (PLEG): container finished" podID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerID="c32b88e92c630ccf9335c658e7f2308464fde0830b6142be5646ae90b5b89a58" exitCode=0 Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.907154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" event={"ID":"88b17b3d-7159-4bd5-b22d-d3eecaca151c","Type":"ContainerDied","Data":"c32b88e92c630ccf9335c658e7f2308464fde0830b6142be5646ae90b5b89a58"} Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.970412 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.970478 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.970517 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.970549 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.909976 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.972300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.910012 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:13:57.972944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.910037 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:13:57.980715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.911156 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "kube-api-access-jg9m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:57.997285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.922862 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:57.997285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.941286 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:57.997285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.941353 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.006265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.959556 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.028502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:57.959625 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.045750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.045705 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.054713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.048881 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:13:58.054713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.048983 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:13:58.054713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.049020 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:13:58.054713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.049056 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:13:58.054713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.049088 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:13:58.060288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.058010 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:58.099547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.064427 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb"] Nov 01 22:13:58.163521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.101026 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" Nov 01 22:13:58.163521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.101057 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" Nov 01 22:13:58.163521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.101272 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" containerName="istio-proxy" Nov 01 22:13:58.234116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.165667 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:13:58.234116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.234047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:58.267481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.237027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") pod \"a39f88cf-9031-4d5e-86fe-233ae699a781\" (UID: \"a39f88cf-9031-4d5e-86fe-233ae699a781\") " Nov 01 22:13:58.296251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.296217 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:13:58.296548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.296514 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" event={"ID":"12da8b2d-cbec-446a-957a-2bd7eae29773","Type":"ContainerStarted","Data":"0cc20956c4426d4e6e7c9238443c9812463fd72ae00ade5184371d00abeddaae"} Nov 01 22:13:58.296794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.296777 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.298334 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:13:58.298225 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a39f88cf-9031-4d5e-86fe-233ae699a781/volumes/kubernetes.io~downward-api/istio-podinfo Nov 01 22:13:58.298334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.298252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo" pod "a39f88cf-9031-4d5e-86fe-233ae699a781" (UID: "a39f88cf-9031-4d5e-86fe-233ae699a781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:58.301344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.301319 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.301475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.301347 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9m9\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-kube-api-access-jg9m9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.301475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.301378 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f88cf-9031-4d5e-86fe-233ae699a781-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.375425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.375385 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" event={"ID":"52e28a5c-aaaa-4698-9c75-62d149c5c19b","Type":"ContainerStarted","Data":"c4c7fc9bed79b066e80489282e1f301176ec8d0600cbcdf4b407bc722b305604"} Nov 01 22:13:58.398339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.378851 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:13:58.398339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.379014 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:13:58.398717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.398698 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 22:13:58.424765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.424724 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.446248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.425392 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:13:58.464729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.427287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ffv\" (UniqueName: \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.471649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.471605 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.641871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.641827 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.643170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.440329 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:13:58.643403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.643368 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:13:58.643609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.521308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:13:58.643876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.489515 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.644620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.493149 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.721668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.721632 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:13:58.738465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.730025 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:13:58.770898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.770375 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.771267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.771241 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.776560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.776525 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.776763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.776745 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.776980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.776964 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.777126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.777108 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.781605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.781581 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.781840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.781815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.787645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.787618 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\") pod \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\" (UID: \"88b17b3d-7159-4bd5-b22d-d3eecaca151c\") " Nov 01 22:13:58.814607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.814563 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:58.815205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.815177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:58.825350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.825317 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.855536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ffv\" (UniqueName: \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.855788 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.855875 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.856080 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.856097 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.856111 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:58.885503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.868561 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.893532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.892131 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.906960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.894128 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.952298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.952261 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:58.968472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.729201 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:13:58.968755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.968717 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:13:58.968968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.968943 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:58.997801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.997753 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:59.058469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.057725 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:13:59.058469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.057849 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:13:59.058469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.057880 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:13:59.058469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.057922 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:13:59.083710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.043566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:13:59.083710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.083534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:13:59.083941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.998545 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:13:59.084014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.998573 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": context deadline exceeded" Nov 01 22:13:59.084120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:58.998506 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.090647 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "kube-api-access-cdkr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.092069 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.107717 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.113687 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkr4\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-kube-api-access-cdkr4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.113718 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.113764 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.113817 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:59.122541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.114333 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:13:59.159478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.150163 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:59.226469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.160699 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:59.254157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.188612 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:59.254157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.191696 5036 cpu_manager.go:414] "containerMap: removing container" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" Nov 01 22:13:59.254157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.250185 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" Nov 01 22:13:59.254157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.250557 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" containerName="istio-proxy" Nov 01 22:13:59.265473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.258674 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:13:59.272467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.265730 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:13:59.313878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.313843 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:13:59.314250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.314227 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb"] Nov 01 22:13:59.339686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.175959 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:59.339913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.168809 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:13:59.356470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.241567 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:59.356470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.343528 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy" pod "88b17b3d-7159-4bd5-b22d-d3eecaca151c" (UID: "88b17b3d-7159-4bd5-b22d-d3eecaca151c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:59.371471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.370031 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.371471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.370065 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.371471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.370083 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.395502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.378052 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.417859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.413917 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:13:59.417859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.413971 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:13:59.734539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.649918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.746468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.736500 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.764612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.734799 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:13:59.790471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.683665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ffv\" (UniqueName: \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\") pod \"maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:13:59.830532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.377674 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:13:59.830801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.830780 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:13:59.830961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.830940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" event={"ID":"88b17b3d-7159-4bd5-b22d-d3eecaca151c","Type":"ContainerDied","Data":"1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0"} Nov 01 22:13:59.831185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.785614 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.831358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.831339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.850027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.831583 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.850278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.850252 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.850492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.850470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.859164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.850647 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6922n\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.863742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.863704 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:13:59.867846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.837337 5036 scope.go:122] "RemoveContainer" containerID="c32b88e92c630ccf9335c658e7f2308464fde0830b6142be5646ae90b5b89a58" Nov 01 22:13:59.892994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.873627 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b17b3d-7159-4bd5-b22d-d3eecaca151c-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:59.893511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.837963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz" event={"ID":"a39f88cf-9031-4d5e-86fe-233ae699a781","Type":"ContainerDied","Data":"44bae5cb506dce92156ee1976d165f38ba493e62a2027d3c1f388fdaaefcd1e5"} Nov 01 22:13:59.900968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.900929 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": context deadline exceeded" Nov 01 22:14:00.143455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.900737 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.270569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.927023 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.270909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.270851 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:00.271085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.024124 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:00.271935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:13:59.984716 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.273191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.148782 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.273372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.273351 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:00.273514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.183974 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:14:00.326501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.245353 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.355348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.353899 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:14:00.356642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.353939 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:14:00.471595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.423362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.353993 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.354014 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.354032 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.353958 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.446022 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.516314 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.607403 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.085526 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.516352 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.155141 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.525879 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.533574 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6922n\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.615802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616237 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616360 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616567 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616885 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.616932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.587246 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.646466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.627767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.671487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.046963 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:14:00.805560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.777917 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.931062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.871344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.931062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.874361 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:14:00.931062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.874822 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.931062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.874982 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.931062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.875821 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:00.967590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.967517 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.967918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.967883 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:14:01.027475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.977768 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.027475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:00.977835 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:14:01.027475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.001086 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.054468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.052010 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.054468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.052070 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:01.054468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.052574 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.054468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.052613 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:01.054468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.053830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6922n\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.167579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.155529 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.175361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.175325 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.265856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.265508 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.267678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.267653 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.293505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.288425 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.510534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.510011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:01.531886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.531796 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:14:01.582056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.582016 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.600573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.582309 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:14:01.609022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.603768 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:14:01.609539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.603954 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.609971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.606113 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.610490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.610463 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:14:01.617349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.617295 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.641163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.641123 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:14:01.642346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.642317 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:14:01.724241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.708331 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:01.921276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.920257 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:01.925353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.925319 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.929634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:01.925387 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:02.001671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.001635 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.002269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.002235 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:02.076489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.064930 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.119016 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:02.118982 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.459s" Nov 01 22:14:02.119243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.119224 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:14:02.119543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.119523 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:14:02.123420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.119734 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:14:02.131595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.131574 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:14:02.235694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.235606 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:14:02.236267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.236236 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:14:02.264554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.264510 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 22:14:02.276591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.276552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.334123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.303121 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.744566 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.745982 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.746193 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.746395 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.746621 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:14:02.749800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.746822 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:14:02.778884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.752551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:14:02.811100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.811054 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.869371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.869331 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.869697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.869660 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:14:02.959475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.959427 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.959722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.959690 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:02.974468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.974127 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.974468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:02.974195 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:03.087470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.081752 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": context deadline exceeded" Nov 01 22:14:03.147268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.138531 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:14:03.147268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.138879 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:14:03.367140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.261795 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:14:03.367140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.292021 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:03.723515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.698629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:14:03.820469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.723544 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 01 22:14:03.820469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.815082 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 01 22:14:03.820469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.724086 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:03.830226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.830181 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:03.830578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.830540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:14:03.830815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.830792 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:14:03.831039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.831013 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:14:03.831249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.831229 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:14:03.843933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.843738 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:03.884807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.883889 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:03.884807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.883977 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:03.986676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.969634 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:14:04.008683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:03.998553 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.167800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.141723 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.171530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.169593 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:14:04.191348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.191291 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 01 22:14:04.237594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.236633 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.237594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.236713 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:04.254021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.253773 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:04.281364 5036 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.521s" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281435 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281616 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281662 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281697 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281735 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:14:04.283673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.281772 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:14:04.329979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.329000 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:04.336703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.336672 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Nov 01 22:14:04.338573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.338422 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 01 22:14:04.380788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.380598 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:14:04.382971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.382896 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:14:04.395732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.395702 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:14:04.398605 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:04.398561 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f88cf_9031_4d5e_86fe_233ae699a781.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b17b3d_7159_4bd5_b22d_d3eecaca151c.slice/crio-1931e382cb0042cab11f79cbef1a071988e0e4a9a8c0b1133d7fd50420cffcd0\": RecentStats: unable to find data in memory cache]" Nov 01 22:14:04.542497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.538144 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:14:04.551029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.550987 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.657058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.657004 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.661092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.661067 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:14:04.661972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.661267 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.662607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.662582 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 01 22:14:04.665233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.665213 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.665393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.665374 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 01 22:14:04.712471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.696770 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:14:04.757073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.756994 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:14:04.779966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.779939 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:14:04.780552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.780531 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:14:04.833391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.833339 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.836747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.833630 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.837548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.833663 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": context deadline exceeded" Nov 01 22:14:04.838233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.833687 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.854625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.848747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:14:04.983900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.982323 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:04.983900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.982422 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:04.983900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.982490 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:04.985468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.985415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.986873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:04.986857 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:14:05.085905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.059205 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:14:05.085905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.073311 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:14:05.093878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.092728 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:14:05.095638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.094063 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:14:05.109026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.108972 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podStartSLOduration=109.108953838 podStartE2EDuration="1m49.108953838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:59.368614445 +0000 UTC m=+1793.061594934" watchObservedRunningTime="2026-11-01 22:14:05.108953838 +0000 UTC m=+1798.801934326" Nov 01 22:14:05.124517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.124379 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:14:05.132037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.132013 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:14:05.267522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.260815 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podStartSLOduration=109.260799997 podStartE2EDuration="1m49.260799997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:00.523785293 +0000 UTC m=+1794.216765780" watchObservedRunningTime="2026-11-01 22:14:05.260799997 +0000 UTC m=+1798.953780550" Nov 01 22:14:05.288747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.274551 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:05.314845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.314815 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 22:14:05.322484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.320955 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-wwrcz"] Nov 01 22:14:05.359698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.359656 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" event={"ID":"ba5d8d2d-2acc-4a60-afa8-178c19c96559","Type":"ContainerStarted","Data":"a264636bc89452ced955c5bef9a305333f5e0446536645ad245d856617bb4a49"} Nov 01 22:14:05.364146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.364111 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:14:05.403684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.387690 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" event={"ID":"02428bf4-f476-4f4d-9847-5668c194869f","Type":"ContainerStarted","Data":"f3990aa2368b75cfc37cf0cde14d0fc80f73b0774b485ee4c5370dcae7df3487"} Nov 01 22:14:05.403684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.387849 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.403684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.393237 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:14:05.558031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.556061 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.558031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.556220 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.558031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.556326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.582824 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.583625 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.583666 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.583793 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.583897 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrqb\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.591518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.583985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.622771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.622663 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb"] Nov 01 22:14:05.672671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.669793 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 22:14:05.678481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.678084 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-mr5tm"] Nov 01 22:14:05.685943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.685907 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.688900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.687101 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.689249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.689226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.689821 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690018 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690120 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690261 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690353 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690392 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.690594 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrqb\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.691795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.691370 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.693194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.692632 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.693194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.693116 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.752417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.752163 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.752417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.752333 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.762062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.762007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.766470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.762925 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrqb\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\") pod \"e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:05.797646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.797618 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:14:05.830390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.827525 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:14:05.830390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.829350 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:14:05.832728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.832705 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:14:05.845057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.844890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:05.879479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.871377 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:14:05.879479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.879469 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:14:05.884776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.884748 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:14:05.985898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:05.985862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:06.034550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.033278 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:06.353534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.353499 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:14:06.354056 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:14:06.354031 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5279c42_55d9_4297_8279_fb4b2d65e4e0.slice/crio-e7b4d65dfed447c8b2e3f3ceb02bfd59afc44eb85be987d5257378507329a9e9 WatchSource:0}: Error finding container e7b4d65dfed447c8b2e3f3ceb02bfd59afc44eb85be987d5257378507329a9e9: Status 404 returned error can't find the container with id e7b4d65dfed447c8b2e3f3ceb02bfd59afc44eb85be987d5257378507329a9e9 Nov 01 22:14:06.359174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.358170 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:06.359174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.358257 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:06.359174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.358299 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:06.475459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.475413 5036 generic.go:356] "Generic (PLEG): container finished" podID="ba5d8d2d-2acc-4a60-afa8-178c19c96559" containerID="4f94c45afa2afeb900cd54bfa85fe6f2137b307641dd90fd59d3a8ab5f4681d3" exitCode=0 Nov 01 22:14:06.633471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.632881 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b17b3d-7159-4bd5-b22d-d3eecaca151c" path="/var/lib/kubelet/pods/88b17b3d-7159-4bd5-b22d-d3eecaca151c/volumes" Nov 01 22:14:06.636060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.635998 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39f88cf-9031-4d5e-86fe-233ae699a781" path="/var/lib/kubelet/pods/a39f88cf-9031-4d5e-86fe-233ae699a781/volumes" Nov 01 22:14:06.636846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.636679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" event={"ID":"e5279c42-55d9-4297-8279-fb4b2d65e4e0","Type":"ContainerStarted","Data":"e7b4d65dfed447c8b2e3f3ceb02bfd59afc44eb85be987d5257378507329a9e9"} Nov 01 22:14:06.636846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.636723 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" event={"ID":"ba5d8d2d-2acc-4a60-afa8-178c19c96559","Type":"ContainerDied","Data":"4f94c45afa2afeb900cd54bfa85fe6f2137b307641dd90fd59d3a8ab5f4681d3"} Nov 01 22:14:06.636846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.636751 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" event={"ID":"02428bf4-f476-4f4d-9847-5668c194869f","Type":"ContainerStarted","Data":"c5db5fd3d10009344d93802d152fc6d0f860adc6d03c2ad4dade50fdf22bfdcf"} Nov 01 22:14:06.670464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.668418 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podStartSLOduration=80.66840422 podStartE2EDuration="1m20.66840422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:06.654012857 +0000 UTC m=+1800.346993346" watchObservedRunningTime="2026-11-01 22:14:06.66840422 +0000 UTC m=+1800.361384725" Nov 01 22:14:06.942406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:06.942324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:07.326521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.326484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:07.375804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.375060 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:14:07.616519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.594839 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:07.803384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.803209 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:07.803384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.803308 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:07.866490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:07.859560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:08.134748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.134709 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" event={"ID":"e5279c42-55d9-4297-8279-fb4b2d65e4e0","Type":"ContainerStarted","Data":"21f4f1f947365569db14c22d4ab809f0400fa68378860e7c3fb59f3b16d0caa7"} Nov 01 22:14:08.273280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.273220 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podStartSLOduration=207.273201322 podStartE2EDuration="3m27.273201322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:08.258953282 +0000 UTC m=+1801.951933766" watchObservedRunningTime="2026-11-01 22:14:08.273201322 +0000 UTC m=+1801.966181826" Nov 01 22:14:08.398535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.389013 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:14:08.668369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.663605 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:08.668369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.663667 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:08.769479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:08.765771 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:09.037464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.036036 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:09.037464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.036134 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:09.039548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.039515 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:09.055776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.055746 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:14:09.176073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.175982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\") pod \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " Nov 01 22:14:09.176248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.176198 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\") pod \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " Nov 01 22:14:09.176400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.176380 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\" (UniqueName: \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\") pod \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\" (UID: \"ba5d8d2d-2acc-4a60-afa8-178c19c96559\") " Nov 01 22:14:09.181143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.181110 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca" pod "ba5d8d2d-2acc-4a60-afa8-178c19c96559" (UID: "ba5d8d2d-2acc-4a60-afa8-178c19c96559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:09.213471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.212812 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv" pod "ba5d8d2d-2acc-4a60-afa8-178c19c96559" (UID: "ba5d8d2d-2acc-4a60-afa8-178c19c96559"). InnerVolumeSpecName "kube-api-access-v9ffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:09.216114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.216071 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca" pod "ba5d8d2d-2acc-4a60-afa8-178c19c96559" (UID: "ba5d8d2d-2acc-4a60-afa8-178c19c96559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:14:09.278475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.273863 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" Nov 01 22:14:09.293468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.280263 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb" event={"ID":"ba5d8d2d-2acc-4a60-afa8-178c19c96559","Type":"ContainerDied","Data":"a264636bc89452ced955c5bef9a305333f5e0446536645ad245d856617bb4a49"} Nov 01 22:14:09.293468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.280331 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a264636bc89452ced955c5bef9a305333f5e0446536645ad245d856617bb4a49" Nov 01 22:14:09.293468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.283047 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:09.293468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.283080 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ffv\" (UniqueName: \"kubernetes.io/projected/ba5d8d2d-2acc-4a60-afa8-178c19c96559-kube-api-access-v9ffv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:09.293468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.283101 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5d8d2d-2acc-4a60-afa8-178c19c96559-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:09.452828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.452739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:09.617682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.580094 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:09.644008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:09.643967 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:09.776887 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:09.776164 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5d8d2d_2acc_4a60_afa8_178c19c96559.slice/crio-a264636bc89452ced955c5bef9a305333f5e0446536645ad245d856617bb4a49\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5d8d2d_2acc_4a60_afa8_178c19c96559.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:14:10.034916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:10.034828 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:10.035194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:10.035161 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:10.363300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:10.363216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:10.564366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:10.559422 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:10.564366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:10.559596 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:11.034903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.034857 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:11.035120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.034908 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:11.249007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.248968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:11.558412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.558378 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:11.558600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.558471 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:11.747488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.721460 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:11.855040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.854964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:11.994471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:11.990387 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:12.057491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:12.037546 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:12.057491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:12.037622 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:12.559095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:12.559059 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:14:12.559292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:12.559137 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 01 22:14:12.777920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:12.777880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:13.035180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.035144 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:13.035376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.035207 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:13.461909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.461825 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:13.595962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.595919 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:13.598153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.598128 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:13.614396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:13.614369 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:14:14.034761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:14.034721 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:14:14.035167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:14.034802 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:14:14.320320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:14.320232 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:15.039424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:15.039389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:15.039949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:15.039928 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:15.040626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:15.040600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:14:15.052398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:15.052369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:15.794214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:15.794181 5036 scope.go:122] "RemoveContainer" containerID="b7cdf54ccd27310c853bfeacbdea8a9ce94ab6347b2fbda5cdd08bc9d8775c72" Nov 01 22:14:16.017259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:16.016737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:16.628395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:16.628353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:17.433773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:17.433736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:17.566779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:17.566744 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:18.333477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:18.333405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:18.860860 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:18.860824 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:14:19.087138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:19.087102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:19.745755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:19.745723 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:20.531003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:20.530961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:21.071674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.071645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:21.230765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.230728 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:21.364851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.364764 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:21.718520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.672566 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb"] Nov 01 22:14:21.733940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.733900 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3577a0-9v5wb"] Nov 01 22:14:21.889499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:21.889208 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:22.010522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:22.010370 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:22.445735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:22.445693 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5d8d2d-2acc-4a60-afa8-178c19c96559" path="/var/lib/kubelet/pods/ba5d8d2d-2acc-4a60-afa8-178c19c96559/volumes" Nov 01 22:14:22.632397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:22.632358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:23.449051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:23.449006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:24.138643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:24.138607 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:24.869631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:24.869597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:25.627799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:25.627757 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:26.327813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:26.327769 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:26.458095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:26.458053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:26.596350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:26.596256 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:27.240960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:27.240921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:28.045322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:28.045282 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:28.898931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:28.898892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:29.152333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:29.152240 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:29.246110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:29.246077 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:14:29.246354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:29.246323 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" containerID="cri-o://e3de9af898709b87e0a01f9f22966c469f36b97dff0022215b123bc858b27a96" gracePeriod=30 Nov 01 22:14:29.285970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:29.285936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:29.992078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:29.992034 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:30.121005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:30.120965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:30.666467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:30.666152 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:30.816471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:30.815389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:31.148676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:31.148636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:31.324207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:31.324170 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:14:31.324454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:31.324419 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:14:31.929825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:31.929788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:32.657686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:32.657645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:33.541809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:33.541768 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:34.121185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.121148 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:34.261206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.261112 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:34.452138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.452051 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:34.933576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.933545 5036 generic.go:356] "Generic (PLEG): container finished" podID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerID="e3de9af898709b87e0a01f9f22966c469f36b97dff0022215b123bc858b27a96" exitCode=0 Nov 01 22:14:34.934134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.934112 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerDied","Data":"e3de9af898709b87e0a01f9f22966c469f36b97dff0022215b123bc858b27a96"} Nov 01 22:14:34.934256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:34.934246 5036 scope.go:122] "RemoveContainer" containerID="7e2eb5fc5e0891f6722c72c15186e591c25d7645b76228f3bd89e5a9ca19b24f" Nov 01 22:14:35.105932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.105904 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:14:35.202044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.201951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202054 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202125 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202277 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202405 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202472 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202523 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202593 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\") pod \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\" (UID: \"5372a0d3-c7f1-4f70-819d-34a95cbdf382\") " Nov 01 22:14:35.202749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.202981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.202956 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.203279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.203255 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:35.203502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.203430 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.203502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.203481 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.203502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.203497 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.204326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.204296 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.205341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.205312 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.207741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.206056 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "kube-api-access-lhcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:35.207741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.207071 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:35.209035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.209005 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:35.210519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.210496 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy" pod "5372a0d3-c7f1-4f70-819d-34a95cbdf382" (UID: "5372a0d3-c7f1-4f70-819d-34a95cbdf382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.305654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305625 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.305654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305657 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.305878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305670 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.305878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305684 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhcx4\" (UniqueName: \"kubernetes.io/projected/5372a0d3-c7f1-4f70-819d-34a95cbdf382-kube-api-access-lhcx4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.305878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305697 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.305878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.305709 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5372a0d3-c7f1-4f70-819d-34a95cbdf382-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.598494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.598408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:35.748020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:35.747981 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:36.090149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.085164 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" event={"ID":"5372a0d3-c7f1-4f70-819d-34a95cbdf382","Type":"ContainerDied","Data":"ad873a66ee03fe9a64cae664b4b22fe77f8664f6a529dadd277c088abf67384b"} Nov 01 22:14:36.090149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.085238 5036 scope.go:122] "RemoveContainer" containerID="e3de9af898709b87e0a01f9f22966c469f36b97dff0022215b123bc858b27a96" Nov 01 22:14:36.090149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.085353 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv" Nov 01 22:14:36.301517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.301456 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:14:36.311463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.307908 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-88jkv"] Nov 01 22:14:36.441467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.441372 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" path="/var/lib/kubelet/pods/5372a0d3-c7f1-4f70-819d-34a95cbdf382/volumes" Nov 01 22:14:36.653037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:36.652994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:37.548612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:37.548577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:38.229107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:38.229073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:38.862861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:38.862821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:38.931616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:38.930513 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:14:39.553192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:39.553155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:40.518010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:40.517969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:41.369878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:41.369826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:42.135296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:42.135261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:42.831540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:42.831459 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:43.689360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:43.689324 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:44.343334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.343296 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:44.691089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.690997 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:14:44.691654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.691303 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" containerID="cri-o://fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82" gracePeriod=30 Nov 01 22:14:44.703048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.701629 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:14:44.703048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.701861 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" containerID="cri-o://79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f" gracePeriod=30 Nov 01 22:14:44.709417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.709377 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:14:44.709674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:44.709625 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" containerID="cri-o://6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4" gracePeriod=30 Nov 01 22:14:45.040139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.040100 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:45.136992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.134970 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:14:45.136992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.135223 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" containerID="cri-o://ef9c3df2af2986ef8a79709c0f177029e1e1c54e0e34a72e7f281afcfc6c3ed0" gracePeriod=30 Nov 01 22:14:45.146351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.146196 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:14:45.146603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.146544 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" containerID="cri-o://837fcc815486c7bdfac99e5fcaa503333072ff0f86410fe5ea300510fe18b831" gracePeriod=30 Nov 01 22:14:45.858259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:45.858222 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:46.547310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:46.547272 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:47.321829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:47.321794 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:48.391932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:48.391885 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:48.860698 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:48.860660 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:14:48.906846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:48.906807 5036 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:14:48.907038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:48.906889 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:14:49.165275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:49.165195 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:14:49.165275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:49.165257 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:14:49.262836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:49.262799 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:49.472045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:49.471969 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 22:14:49.472465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:49.472242 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" containerID="cri-o://d3342f0d5d5a08dfc838b0a96c4692acae635f9b1c94c4dd738591eb6ec0f8e0" gracePeriod=30 Nov 01 22:14:50.139487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.133028 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:50.237893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.237320 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:14:50.241315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.241219 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:14:50.341420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.341388 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345241 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345280 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345366 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345492 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345584 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345650 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345710 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345751 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345777 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345893 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.345980 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346045 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346082 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346291 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.346387 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.347603 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\") pod \"54b6e3b1-0b63-4544-8c85-553a784350ee\" (UID: \"54b6e3b1-0b63-4544-8c85-553a784350ee\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.347644 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\") pod \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\" (UID: \"35968bb6-3e12-47ba-a9e3-a5a86801a0ef\") " Nov 01 22:14:50.348904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.347955 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "kube-api-access-m8rz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.348925 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.348949 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.348964 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rz6\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-kube-api-access-m8rz6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.348979 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349491 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.349971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:50.350026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.350015 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.350618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.350306 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:50.352431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.352246 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:50.352431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.352282 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "kube-api-access-vhhwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.353726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.353687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:50.354001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.353966 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.354001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.353978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token" pod "54b6e3b1-0b63-4544-8c85-553a784350ee" (UID: "54b6e3b1-0b63-4544-8c85-553a784350ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.355751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.355714 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token" pod "35968bb6-3e12-47ba-a9e3-a5a86801a0ef" (UID: "35968bb6-3e12-47ba-a9e3-a5a86801a0ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.450047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450005 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450116 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450159 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450330 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450562 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450604 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.450745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.450731 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\") pod \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\" (UID: \"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f\") " Nov 01 22:14:50.451748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.451715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.451990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.451967 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.452372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452351 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452376 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452386 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452397 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452410 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452420 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452439 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6e3b1-0b63-4544-8c85-553a784350ee-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452473 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452487 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452508 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452525 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452541 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452556 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452571 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhwz\" (UniqueName: \"kubernetes.io/projected/54b6e3b1-0b63-4544-8c85-553a784350ee-kube-api-access-vhhwz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452584 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6e3b1-0b63-4544-8c85-553a784350ee-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452602 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35968bb6-3e12-47ba-a9e3-a5a86801a0ef-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.452896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.452855 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:50.453395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.453136 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.455028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.454998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:50.455188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.455172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.458238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.458207 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.458857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.458828 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.462222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.462193 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp" pod "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" (UID: "5c7bca4b-42cf-4bcf-a74c-c6947cb6072f"). InnerVolumeSpecName "kube-api-access-d45pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.518524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.518271 5036 generic.go:356] "Generic (PLEG): container finished" podID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerID="837fcc815486c7bdfac99e5fcaa503333072ff0f86410fe5ea300510fe18b831" exitCode=0 Nov 01 22:14:50.518524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.518408 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" event={"ID":"8eac855d-ad84-4737-b0ad-bf2dea25eefd","Type":"ContainerDied","Data":"837fcc815486c7bdfac99e5fcaa503333072ff0f86410fe5ea300510fe18b831"} Nov 01 22:14:50.535496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.535149 5036 generic.go:356] "Generic (PLEG): container finished" podID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerID="6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4" exitCode=0 Nov 01 22:14:50.535690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.535493 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" Nov 01 22:14:50.537112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.536254 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" event={"ID":"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f","Type":"ContainerDied","Data":"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4"} Nov 01 22:14:50.537112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.536316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng" event={"ID":"5c7bca4b-42cf-4bcf-a74c-c6947cb6072f","Type":"ContainerDied","Data":"e19543285aa909bbafde0a047d762cf0ad8f4e80ac6ad5522dcdf6899507307f"} Nov 01 22:14:50.537112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.536347 5036 scope.go:122] "RemoveContainer" containerID="6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4" Nov 01 22:14:50.542316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.542292 5036 generic.go:356] "Generic (PLEG): container finished" podID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerID="79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f" exitCode=0 Nov 01 22:14:50.542426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.542400 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerDied","Data":"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f"} Nov 01 22:14:50.542679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.542660 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" event={"ID":"35968bb6-3e12-47ba-a9e3-a5a86801a0ef","Type":"ContainerDied","Data":"c0ab7636c3508e6e6057d4fd81dafeea6665963fa578456cd436771150c82226"} Nov 01 22:14:50.542792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.542776 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm" Nov 01 22:14:50.546462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.546300 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:14:50.547582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.547217 5036 generic.go:356] "Generic (PLEG): container finished" podID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerID="ef9c3df2af2986ef8a79709c0f177029e1e1c54e0e34a72e7f281afcfc6c3ed0" exitCode=0 Nov 01 22:14:50.547582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.547296 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" event={"ID":"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49","Type":"ContainerDied","Data":"ef9c3df2af2986ef8a79709c0f177029e1e1c54e0e34a72e7f281afcfc6c3ed0"} Nov 01 22:14:50.550853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.549380 5036 generic.go:356] "Generic (PLEG): container finished" podID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerID="fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82" exitCode=0 Nov 01 22:14:50.550853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.549550 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" Nov 01 22:14:50.550853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.549568 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" event={"ID":"54b6e3b1-0b63-4544-8c85-553a784350ee","Type":"ContainerDied","Data":"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82"} Nov 01 22:14:50.550853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.549620 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg" event={"ID":"54b6e3b1-0b63-4544-8c85-553a784350ee","Type":"ContainerDied","Data":"92ce322e2bc39e7686ab2f2eb061bdb25889ae9792978fd0fccb1359d0277e11"} Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554618 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554647 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554662 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554677 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d45pp\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-kube-api-access-d45pp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554693 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554707 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.555395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.554720 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.556484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.556422 5036 scope.go:122] "RemoveContainer" containerID="6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4" Nov 01 22:14:50.556776 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:50.556744 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4\": container with ID starting with 6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4 not found: ID does not exist" containerID="6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4" Nov 01 22:14:50.556882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.556785 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4"} err="failed to get container status \"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4\": rpc error: code = NotFound desc = could not find container \"6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4\": container with ID starting with 6187c71dcb16527a5f95c0b0762f82f2bb5c7e9a6f62bed40d7691c1786111c4 not found: ID does not exist" Nov 01 22:14:50.556882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.556810 5036 scope.go:122] "RemoveContainer" containerID="79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f" Nov 01 22:14:50.562272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.562238 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:14:50.567854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.567831 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-tzkng"] Nov 01 22:14:50.573918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.573619 5036 scope.go:122] "RemoveContainer" containerID="b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1" Nov 01 22:14:50.583622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.580996 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:14:50.587662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.587637 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6grvg"] Nov 01 22:14:50.594032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.594007 5036 scope.go:122] "RemoveContainer" containerID="79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f" Nov 01 22:14:50.594365 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:50.594339 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f\": container with ID starting with 79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f not found: ID does not exist" containerID="79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f" Nov 01 22:14:50.594524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.594375 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f"} err="failed to get container status \"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f\": rpc error: code = NotFound desc = could not find container \"79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f\": container with ID starting with 79e207a68b307d8e7939e64de0917baf6dee94ae736a2d56c4fe35b95c25435f not found: ID does not exist" Nov 01 22:14:50.594524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.594397 5036 scope.go:122] "RemoveContainer" containerID="b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1" Nov 01 22:14:50.594760 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:50.594738 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1\": container with ID starting with b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1 not found: ID does not exist" containerID="b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1" Nov 01 22:14:50.594884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.594764 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1"} err="failed to get container status \"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1\": rpc error: code = NotFound desc = could not find container \"b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1\": container with ID starting with b68981af63127f5b88bbb78173c64b486d6bfd227bc2b9cbb51b6e05c32167f1 not found: ID does not exist" Nov 01 22:14:50.594884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.594778 5036 scope.go:122] "RemoveContainer" containerID="ef9c3df2af2986ef8a79709c0f177029e1e1c54e0e34a72e7f281afcfc6c3ed0" Nov 01 22:14:50.598922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.598902 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:14:50.607217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.607198 5036 scope.go:122] "RemoveContainer" containerID="fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82" Nov 01 22:14:50.626600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.626577 5036 scope.go:122] "RemoveContainer" containerID="fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82" Nov 01 22:14:50.632790 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:50.632644 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82\": container with ID starting with fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82 not found: ID does not exist" containerID="fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82" Nov 01 22:14:50.632790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.632711 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82"} err="failed to get container status \"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82\": rpc error: code = NotFound desc = could not find container \"fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82\": container with ID starting with fc8de284784a70781fa11e7f078735618eb3b66d72bd50d70ea0da8851794b82 not found: ID does not exist" Nov 01 22:14:50.643703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.643680 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:14:50.649221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.649194 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-n9fwm"] Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657352 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657492 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657625 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657656 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657707 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.657749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657752 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657847 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657927 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657945 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.657968 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658007 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658068 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658115 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658185 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658220 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658258 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\") pod \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\" (UID: \"8eac855d-ad84-4737-b0ad-bf2dea25eefd\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.658343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\") pod \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\" (UID: \"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49\") " Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.659565 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.659590 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.659858 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.660574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.660903 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.660992 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.662076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.661602 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.663328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.663252 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "kube-api-access-7msrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.665104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.664574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.665104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.664616 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.665104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.664781 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.665104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.664949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:50.665104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.664963 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:50.667150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.667125 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:50.668388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.668365 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:50.668570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.668394 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.669834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.669809 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn" pod "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" (UID: "90a9c4b8-ce31-41ac-a3b4-e754a10fbf49"). InnerVolumeSpecName "kube-api-access-ll6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.670295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.670166 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token" pod "8eac855d-ad84-4737-b0ad-bf2dea25eefd" (UID: "8eac855d-ad84-4737-b0ad-bf2dea25eefd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:50.761172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761137 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761175 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761190 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761206 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761220 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761234 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761247 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761260 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761274 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761287 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761302 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.761320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761316 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eac855d-ad84-4737-b0ad-bf2dea25eefd-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761331 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6gn\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-kube-api-access-ll6gn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761345 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761359 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761372 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761385 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msrj\" (UniqueName: \"kubernetes.io/projected/8eac855d-ad84-4737-b0ad-bf2dea25eefd-kube-api-access-7msrj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.772659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.761399 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:50.893350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:50.893312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:51.407897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.407860 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:51.564552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.564509 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" event={"ID":"90a9c4b8-ce31-41ac-a3b4-e754a10fbf49","Type":"ContainerDied","Data":"57c6776153b11ce4f89e3b11f63fcfb9347604a1b14eec9bf952d6b8b50f3dd3"} Nov 01 22:14:51.564552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.564547 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz" Nov 01 22:14:51.567658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.567631 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" Nov 01 22:14:51.567831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.567634 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq" event={"ID":"8eac855d-ad84-4737-b0ad-bf2dea25eefd","Type":"ContainerDied","Data":"d097ccdd80712da8dedb01de7510e9241996d468f3f78b8451c6b83d91ec5757"} Nov 01 22:14:51.567831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.567777 5036 scope.go:122] "RemoveContainer" containerID="837fcc815486c7bdfac99e5fcaa503333072ff0f86410fe5ea300510fe18b831" Nov 01 22:14:51.594271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.594245 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:14:51.599378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.599351 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwwkz"] Nov 01 22:14:51.610090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.610068 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:14:51.616747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.616545 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-z4dnq"] Nov 01 22:14:51.666922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.666856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:51.847810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:51.847767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.203977 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205139 5036 cpu_manager.go:414] "containerMap: removing container" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205159 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205194 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ba5d8d2d-2acc-4a60-afa8-178c19c96559" containerName="revoke-keys" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205203 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5d8d2d-2acc-4a60-afa8-178c19c96559" containerName="revoke-keys" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205220 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205229 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205255 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205263 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205293 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205302 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205322 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.206000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.205330 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211398 5036 cpu_manager.go:414] "containerMap: removing container" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211420 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211458 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211468 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211484 5036 cpu_manager.go:414] "containerMap: removing container" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211492 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211735 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211757 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5d8d2d-2acc-4a60-afa8-178c19c96559" containerName="revoke-keys" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211769 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211785 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211800 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211814 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5372a0d3-c7f1-4f70-819d-34a95cbdf382" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211824 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211835 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.212473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.211843 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" containerName="istio-proxy" Nov 01 22:14:52.244908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.238677 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.258595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.258566 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:14:52.267268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.267240 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:52.269803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.269778 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6b4c49-openshift-default-dockercfg-9prbz\"" Nov 01 22:14:52.289015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.288985 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289374 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289493 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289571 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289606 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9rw\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289736 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289785 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.289874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.289841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394500 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394563 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394599 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9rw\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394767 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394815 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394876 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.394969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.399465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.395729 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.400186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.400155 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.441487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.409842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.441487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.410512 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.441487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.410764 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.441487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.411340 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.446259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.445682 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.447329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.447252 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:52.461100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.460981 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.487229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.487195 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9rw\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.554598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.554558 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35968bb6-3e12-47ba-a9e3-a5a86801a0ef" path="/var/lib/kubelet/pods/35968bb6-3e12-47ba-a9e3-a5a86801a0ef/volumes" Nov 01 22:14:52.555316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.555291 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b6e3b1-0b63-4544-8c85-553a784350ee" path="/var/lib/kubelet/pods/54b6e3b1-0b63-4544-8c85-553a784350ee/volumes" Nov 01 22:14:52.574286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.574177 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:52.575135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.574584 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7bca4b-42cf-4bcf-a74c-c6947cb6072f" path="/var/lib/kubelet/pods/5c7bca4b-42cf-4bcf-a74c-c6947cb6072f/volumes" Nov 01 22:14:52.576216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.576191 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eac855d-ad84-4737-b0ad-bf2dea25eefd" path="/var/lib/kubelet/pods/8eac855d-ad84-4737-b0ad-bf2dea25eefd/volumes" Nov 01 22:14:52.577280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.577258 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a9c4b8-ce31-41ac-a3b4-e754a10fbf49" path="/var/lib/kubelet/pods/90a9c4b8-ce31-41ac-a3b4-e754a10fbf49/volumes" Nov 01 22:14:52.723364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:52.723314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:53.279371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.279336 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:14:53.280525 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:14:53.280493 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25475b42_d313_4ab9_9a89_f56cfb3028f6.slice/crio-776a3879d1ae033d6450da59ec1ba5f21e4e28c65c8f3e210134ef782439dfce WatchSource:0}: Error finding container 776a3879d1ae033d6450da59ec1ba5f21e4e28c65c8f3e210134ef782439dfce: Status 404 returned error can't find the container with id 776a3879d1ae033d6450da59ec1ba5f21e4e28c65c8f3e210134ef782439dfce Nov 01 22:14:53.284619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.284574 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:53.284754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.284665 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:53.284754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.284711 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:14:53.530466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.526067 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:53.723606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.723501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerStarted","Data":"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313"} Nov 01 22:14:53.724080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.723885 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerStarted","Data":"776a3879d1ae033d6450da59ec1ba5f21e4e28c65c8f3e210134ef782439dfce"} Nov 01 22:14:53.760346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:53.760285 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podStartSLOduration=1.760267083 podStartE2EDuration="1.760267083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:53.754062768 +0000 UTC m=+1847.447043258" watchObservedRunningTime="2026-11-01 22:14:53.760267083 +0000 UTC m=+1847.453247574" Nov 01 22:14:54.274683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.274643 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:14:54.312533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.312179 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.313988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.313958 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:14:54.325692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.320485 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-d6e25f-serving-cert\"" Nov 01 22:14:54.363481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.363067 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:54.455188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.441910 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.462550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.462500 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.477480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.472844 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvkv\" (UniqueName: \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.485111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.481607 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.535534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.534346 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:54.575418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.575389 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.583533 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.583749 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvkv\" (UniqueName: \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.583795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.584024 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:54.584120 5036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-serving-cert: secret "maas-api-e2e-shared-a-w6-d6e25f-serving-cert" not found Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:14:54.584243 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls podName:3960c33e-59ee-461b-91ce-9cd2a6ab32ac nodeName:}" failed. No retries permitted until 2026-11-01 22:14:55.08421922 +0000 UTC m=+1848.777199699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls") pod "maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" (UID: "3960c33e-59ee-461b-91ce-9cd2a6ab32ac") : secret "maas-api-e2e-shared-a-w6-d6e25f-serving-cert" not found Nov 01 22:14:54.584495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.584491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.584981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.584552 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.587100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.587074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.598160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.598094 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvkv\" (UniqueName: \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:54.750281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.749894 5036 generic.go:356] "Generic (PLEG): container finished" podID="3fea9e49-362e-4da9-a509-4f051892f91c" containerID="d3342f0d5d5a08dfc838b0a96c4692acae635f9b1c94c4dd738591eb6ec0f8e0" exitCode=0 Nov 01 22:14:54.750281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.750092 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" event={"ID":"3fea9e49-362e-4da9-a509-4f051892f91c","Type":"ContainerDied","Data":"d3342f0d5d5a08dfc838b0a96c4692acae635f9b1c94c4dd738591eb6ec0f8e0"} Nov 01 22:14:54.750829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.750691 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:54.760210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:54.751961 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:14:55.012366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.012334 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 22:14:55.098478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.098396 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.098769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.098540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:55.098769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.098655 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.098769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.098724 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.099029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.098915 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.099087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.099045 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.099143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.099111 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.099300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.099191 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.099106 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.100555 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.100764 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.101034 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.101207 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\") pod \"3fea9e49-362e-4da9-a509-4f051892f91c\" (UID: \"3fea9e49-362e-4da9-a509-4f051892f91c\") " Nov 01 22:14:55.102039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102001 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:55.102468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102388 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:55.102723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:55.102723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102712 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:55.102897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102869 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.102897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102886 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.103021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102904 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.103021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102921 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fea9e49-362e-4da9-a509-4f051892f91c-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.103021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102936 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.103021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.102952 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.105426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.105398 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "kube-api-access-r59xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:55.105746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.105665 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:55.105845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.105796 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy" pod "3fea9e49-362e-4da9-a509-4f051892f91c" (UID: "3fea9e49-362e-4da9-a509-4f051892f91c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:55.107566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.107543 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:55.307141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.257328 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:55.307141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.263766 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fea9e49-362e-4da9-a509-4f051892f91c-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.307141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.263798 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59xs\" (UniqueName: \"kubernetes.io/projected/3fea9e49-362e-4da9-a509-4f051892f91c-kube-api-access-r59xs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.307141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.263815 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fea9e49-362e-4da9-a509-4f051892f91c-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:55.551123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.550807 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:55.763745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.763503 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" event={"ID":"3fea9e49-362e-4da9-a509-4f051892f91c","Type":"ContainerDied","Data":"3ff7ff91a64d54813eac958cad2407fa8485fbca0a71ebea5ffc1ce17626793b"} Nov 01 22:14:55.763745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.763591 5036 scope.go:122] "RemoveContainer" containerID="d3342f0d5d5a08dfc838b0a96c4692acae635f9b1c94c4dd738591eb6ec0f8e0" Nov 01 22:14:55.764586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.764556 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4" Nov 01 22:14:55.816551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.816521 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:14:55.825413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.825385 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 22:14:55.831795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:55.831770 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-ddfc4"] Nov 01 22:14:56.089419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.089386 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:56.283855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.283761 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:56.441804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.441713 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" path="/var/lib/kubelet/pods/3fea9e49-362e-4da9-a509-4f051892f91c/volumes" Nov 01 22:14:56.546277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.546233 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:56.775503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.775468 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" event={"ID":"3960c33e-59ee-461b-91ce-9cd2a6ab32ac","Type":"ContainerStarted","Data":"56c18eefb5eea0e82ab86d82bbff3f6b13355fa2d2792ff3b30eafa872f8dd6e"} Nov 01 22:14:56.775962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.775538 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:14:56.775962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.775566 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" event={"ID":"3960c33e-59ee-461b-91ce-9cd2a6ab32ac","Type":"ContainerStarted","Data":"49f05d86d7ed769c2e17de374f0a038f2c2547f5d6146acc460dd2c6ccbba34c"} Nov 01 22:14:56.802792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:56.802744 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" podStartSLOduration=2.434851595 podStartE2EDuration="2.802730709s" totalImagesPullingTime="367.879114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:56.800635353 +0000 UTC m=+1850.493615840" watchObservedRunningTime="2026-11-01 22:14:56.802730709 +0000 UTC m=+1850.495711195" Nov 01 22:14:57.251683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:57.251597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:57.394175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:57.393455 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:57.556944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:57.544523 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:58.337658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:58.337612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:58.990642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:58.990603 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:59.730605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:59.730562 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:14:59.868761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:14:59.868691 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:00.140712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.140678 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:00.141492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.141473 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" Nov 01 22:15:00.141492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.141493 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" Nov 01 22:15:00.141684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.141674 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fea9e49-362e-4da9-a509-4f051892f91c" containerName="istio-proxy" Nov 01 22:15:00.145418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.145399 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:00.145561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.145532 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.150948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.150928 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:00.151060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.151037 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.158801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.156393 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:15:00.158801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.158690 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:00.159480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.159455 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.165127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.164997 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:00.169486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.169467 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.175064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.175032 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.175627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.175608 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:00.183137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.183113 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:00.192214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.191598 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:00.197356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.197337 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:00.203283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203253 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppmv\" (UniqueName: \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.203402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.203491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203432 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58qk\" (UniqueName: \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.203605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203498 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.203605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203598 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.203737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203659 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbksf\" (UniqueName: \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.203737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203700 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.203866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.203805 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzh7f\" (UniqueName: \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.204049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.204029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.204140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.204082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.210904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.210884 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:00.211579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.211541 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.230350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.230328 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.240723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.240694 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:00.259306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.246561 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.259306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.249159 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:00.268977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.268811 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.269183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.269166 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:00.286200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.286165 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:00.286633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.286598 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:00.287324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.286485 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305547 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xn9k\" (UniqueName: \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305613 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305655 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305712 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58qk\" (UniqueName: \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305750 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305783 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjms\" (UniqueName: \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305849 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.305955 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306021 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbksf\" (UniqueName: \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306066 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzh7f\" (UniqueName: \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306277 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306348 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306386 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwh2\" (UniqueName: \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306530 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppmv\" (UniqueName: \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.306576 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqhc\" (UniqueName: \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.307654 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.308597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.309459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.309279 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.310714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.310569 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.323496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.311212 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.341282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.341150 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\") pod \"maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.348405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.347633 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:00.352400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.352376 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbksf\" (UniqueName: \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\") pod \"maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.354859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.354838 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppmv\" (UniqueName: \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\") pod \"maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.356078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.356056 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58qk\" (UniqueName: \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\") pod \"maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.357190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.356743 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:00.358335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.358316 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzh7f\" (UniqueName: \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\") pod \"maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.365664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.365621 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:00.372898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.372874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:00.380627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.380578 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:00.398702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.397040 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:00.406172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.406058 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:00.406563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.406472 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.408326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.408302 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.410246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.410225 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwh2\" (UniqueName: \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.412228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.412207 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxqb\" (UniqueName: \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.412855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.411992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.413089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.413070 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqhc\" (UniqueName: \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.413388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.413366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xn9k\" (UniqueName: \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.413577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.413556 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.413726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.413708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.413909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.413891 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjms\" (UniqueName: \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.414078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.414061 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.414200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.414183 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.416012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.415986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.416310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.416274 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.416595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.416575 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.417255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.417233 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.420860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.420838 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:00.425676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.425650 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwh2\" (UniqueName: \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\") pod \"maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.433605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.433584 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xn9k\" (UniqueName: \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.437403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.437380 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqhc\" (UniqueName: \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.447186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.447159 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:00.447293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.447215 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:00.456532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.456508 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.459655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.457955 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjms\" (UniqueName: \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\") pod \"maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.470531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.470360 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:00.481809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.481786 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:00.485374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.485351 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:00.490597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.490568 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:00.492515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.492496 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.492961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.492943 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:00.500841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.500825 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:00.501070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.501016 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.516730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.516704 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.516836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.516808 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wbk\" (UniqueName: \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.520003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.519925 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.520118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520042 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxqb\" (UniqueName: \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.520118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520076 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv64k\" (UniqueName: \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.520238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfrn\" (UniqueName: \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.520300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520271 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.520368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520301 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwzr\" (UniqueName: \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.520368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.520518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.520427 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.522808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.522775 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.525296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.525022 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:00.531482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.531426 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:00.533941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.533891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxqb\" (UniqueName: \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.534625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.534595 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:00.562028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.560725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:00.595252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.594011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.621919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622021 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv64k\" (UniqueName: \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622108 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfjx\" (UniqueName: \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622147 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfrn\" (UniqueName: \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622208 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622262 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwzr\" (UniqueName: \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622400 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.622493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wbk\" (UniqueName: \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.623617 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.625725 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.626148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.626047 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.631241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.631032 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.660193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.659702 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwzr\" (UniqueName: \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.665485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.663780 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfrn\" (UniqueName: \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.665485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.665217 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:00.674251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.674216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:00.716126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.715875 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:00.724925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.724789 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:00.736894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.736832 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv64k\" (UniqueName: \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.744641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.744604 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wbk\" (UniqueName: \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\") pod \"maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.750468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.750421 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfjx\" (UniqueName: \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.750588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.750535 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.751663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.751633 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.791351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.791275 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfjx\" (UniqueName: \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\") pod \"maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:00.798977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.798755 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:00.800241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.800184 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:00.835362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.835124 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:00.847298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.847094 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:00.859246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.859064 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:00.871650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:00.871465 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:01.233518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.233492 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:01.238267 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:01.238237 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1448ce4_6179_435e_b9fc_55a5e6b5a7c9.slice/crio-8c1164b96e9f2f66fe26a41ca38f3af4fa5bee39e0be151d817a2203106c4552 WatchSource:0}: Error finding container 8c1164b96e9f2f66fe26a41ca38f3af4fa5bee39e0be151d817a2203106c4552: Status 404 returned error can't find the container with id 8c1164b96e9f2f66fe26a41ca38f3af4fa5bee39e0be151d817a2203106c4552 Nov 01 22:15:01.239361 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:01.239336 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f13ed8_24f7_493b_8c7f_a8d632156d69.slice/crio-a84eb30a6d9c08b1ae7a01408ccfb02d5e5adc88bd3552038325b3309d5a28a4 WatchSource:0}: Error finding container a84eb30a6d9c08b1ae7a01408ccfb02d5e5adc88bd3552038325b3309d5a28a4: Status 404 returned error can't find the container with id a84eb30a6d9c08b1ae7a01408ccfb02d5e5adc88bd3552038325b3309d5a28a4 Nov 01 22:15:01.240247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.240224 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:01.340382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.340356 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:01.343676 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:01.343646 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be5bd1f_7a22_44ec_92f8_f156941b97c2.slice/crio-858040f4a2151ef774ef2816f930ac32c6ff0609d642a460dbcbd028c47b82f9 WatchSource:0}: Error finding container 858040f4a2151ef774ef2816f930ac32c6ff0609d642a460dbcbd028c47b82f9: Status 404 returned error can't find the container with id 858040f4a2151ef774ef2816f930ac32c6ff0609d642a460dbcbd028c47b82f9 Nov 01 22:15:01.357504 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:01.357463 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75baa909_f44b_4da4_9756_0dc10f2ad0f6.slice/crio-772651eca52100ca0b6e5619575b55d040df7e645b99f3cc3148b4f3445c9a40 WatchSource:0}: Error finding container 772651eca52100ca0b6e5619575b55d040df7e645b99f3cc3148b4f3445c9a40: Status 404 returned error can't find the container with id 772651eca52100ca0b6e5619575b55d040df7e645b99f3cc3148b4f3445c9a40 Nov 01 22:15:01.364197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.363562 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:01.372339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.372309 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:01.447649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.447615 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:01.553920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.553863 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:01.675930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.675902 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:01.705573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.705534 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:01.714311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.714290 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:01.722914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.722891 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:01.730510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.730482 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:01.733981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.733946 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:01.738832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.738809 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:01.743632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.743613 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:01.748143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.748121 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:01.900569 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:01.900510 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699f6c9b_412f_4987_85e9_3308e01ae622.slice/crio-4f6e9898ee5b931ca6e5826822912c41ec36eb51a6afad173c5987c694882dc6 WatchSource:0}: Error finding container 4f6e9898ee5b931ca6e5826822912c41ec36eb51a6afad173c5987c694882dc6: Status 404 returned error can't find the container with id 4f6e9898ee5b931ca6e5826822912c41ec36eb51a6afad173c5987c694882dc6 Nov 01 22:15:01.903000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.902971 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerStarted","Data":"8c1164b96e9f2f66fe26a41ca38f3af4fa5bee39e0be151d817a2203106c4552"} Nov 01 22:15:01.906200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.906172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerStarted","Data":"a991f3a27ea0b4d316f130ae54bc878e8cc90260c02dd1f00664d03fc708dbf7"} Nov 01 22:15:01.909742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.908710 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerStarted","Data":"19f9d540713553c07218e87d2cf508412ba3663ee1c5483a92dd04e3b80c6345"} Nov 01 22:15:01.911073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.911040 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerStarted","Data":"07a2ab4ae00aa983c42cac6603ec8825c48f54e7296cbc0c03a7819b82363c80"} Nov 01 22:15:01.915233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.915192 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerStarted","Data":"8c5c68f5a4ce7758f8aefd78230ef2dd23dd539db1071807caa04a3c3084296b"} Nov 01 22:15:01.917302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.917242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerStarted","Data":"772651eca52100ca0b6e5619575b55d040df7e645b99f3cc3148b4f3445c9a40"} Nov 01 22:15:01.921705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.921641 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerStarted","Data":"db881e8bec0b6237f7e623bb5d06afad504af0864fb7db9c219946098272b856"} Nov 01 22:15:01.923943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.923917 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerStarted","Data":"f4c5ad9ec631188dbab957ca7644173ccbc8d9453dfc74c9c82f18958f88020f"} Nov 01 22:15:01.925817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.925790 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerStarted","Data":"4bcadf24fb96c96df16abf3db400df09e2cfc33676d14cc14d6174d1152d64ea"} Nov 01 22:15:01.928714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.928684 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerStarted","Data":"a84eb30a6d9c08b1ae7a01408ccfb02d5e5adc88bd3552038325b3309d5a28a4"} Nov 01 22:15:01.930354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.930333 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerStarted","Data":"4fff79e15fa0a0cd0f6a9e2a554d294712982aa995ef08466e0d49bbe32f3662"} Nov 01 22:15:01.931838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:01.931816 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerStarted","Data":"858040f4a2151ef774ef2816f930ac32c6ff0609d642a460dbcbd028c47b82f9"} Nov 01 22:15:02.227907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.227870 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:02.839607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.839581 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:15:02.932742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.932706 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:02.950760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.950426 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerStarted","Data":"1fe289902fb03904f42cda0ef6dd93b5a97a4f794ffc805724d25ea328a4e15b"} Nov 01 22:15:02.952798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.952766 5036 generic.go:356] "Generic (PLEG): container finished" podID="86fc72a8-060e-4573-b43c-50c40521abf0" containerID="0947b4a75d6f3cac283019a69649aaca7d05bbda7fc5c35babd6e54bb3c285a5" exitCode=60 Nov 01 22:15:02.952937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.952866 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerDied","Data":"0947b4a75d6f3cac283019a69649aaca7d05bbda7fc5c35babd6e54bb3c285a5"} Nov 01 22:15:02.954058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.954035 5036 scope.go:122] "RemoveContainer" containerID="0947b4a75d6f3cac283019a69649aaca7d05bbda7fc5c35babd6e54bb3c285a5" Nov 01 22:15:02.955599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.955570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerStarted","Data":"0ac048d1eaf4c4c350a4cbd27130dc8a52f60c4c9649e396583a128a1fdb9b30"} Nov 01 22:15:02.963917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:02.962654 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerStarted","Data":"7b962287852485121408b8ccf63936492b87b2de7028a8c17c591e5600c5cff0"} Nov 01 22:15:03.032760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.023405 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" podStartSLOduration=3.023385555 podStartE2EDuration="3.023385555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.022139612 +0000 UTC m=+1856.715120102" watchObservedRunningTime="2026-11-01 22:15:03.023385555 +0000 UTC m=+1856.716366041" Nov 01 22:15:03.103920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.097172 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerStarted","Data":"6a5aaa8b33091c2cb8c5ffe697139f9329e1b824581f10cc918074efe0a51b5e"} Nov 01 22:15:03.125317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.124953 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerStarted","Data":"38d9685860a9f28699a24cb4fbcd02a17ed306a69214ea54d776d9580efca58f"} Nov 01 22:15:03.125317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.125277 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" podStartSLOduration=3.125258024 podStartE2EDuration="3.125258024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.048216026 +0000 UTC m=+1856.741196514" watchObservedRunningTime="2026-11-01 22:15:03.125258024 +0000 UTC m=+1856.818238511" Nov 01 22:15:03.134561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.134518 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerStarted","Data":"4f6e9898ee5b931ca6e5826822912c41ec36eb51a6afad173c5987c694882dc6"} Nov 01 22:15:03.139230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.138926 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerStarted","Data":"e55810f0a445f4529abcfa42de7301080b681b27eac48be4669902e0637a248c"} Nov 01 22:15:03.139736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.139681 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" podStartSLOduration=3.139664319 podStartE2EDuration="3.139664319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.136613428 +0000 UTC m=+1856.829593909" watchObservedRunningTime="2026-11-01 22:15:03.139664319 +0000 UTC m=+1856.832644804" Nov 01 22:15:03.141720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.141694 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerStarted","Data":"3f2881a350bd48eb6801969e617f8b44c78be56d93c5146de41fc9d56be9a84e"} Nov 01 22:15:03.144848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.144827 5036 generic.go:356] "Generic (PLEG): container finished" podID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerID="2a89e6e8f5d7f469a2dc75a685ae4c2e768409d38db51c3e9d216bceb2431b65" exitCode=60 Nov 01 22:15:03.144941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.144907 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerDied","Data":"2a89e6e8f5d7f469a2dc75a685ae4c2e768409d38db51c3e9d216bceb2431b65"} Nov 01 22:15:03.145144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.145126 5036 scope.go:122] "RemoveContainer" containerID="2a89e6e8f5d7f469a2dc75a685ae4c2e768409d38db51c3e9d216bceb2431b65" Nov 01 22:15:03.147078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.147055 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerStarted","Data":"e5b4042fa7422a8ebd679c3692a929e8bd5c4c339940288fb4ee179c816329bc"} Nov 01 22:15:03.148960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.148936 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerStarted","Data":"69d276e634564839675997ae36ac70612297ae9fd092b3be7a40cb48267aebaf"} Nov 01 22:15:03.150966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.150943 5036 generic.go:356] "Generic (PLEG): container finished" podID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerID="be843eb6b7874ee5b6ad6478835b0cd6c1d4d2903c74f5982030fe561a057481" exitCode=60 Nov 01 22:15:03.151107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.151044 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerDied","Data":"be843eb6b7874ee5b6ad6478835b0cd6c1d4d2903c74f5982030fe561a057481"} Nov 01 22:15:03.151207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.151190 5036 scope.go:122] "RemoveContainer" containerID="be843eb6b7874ee5b6ad6478835b0cd6c1d4d2903c74f5982030fe561a057481" Nov 01 22:15:03.152747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.152729 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" event={"ID":"380a1985-8962-4447-9d6a-4a399646e5eb","Type":"ContainerStarted","Data":"4385fa3d8ed4489d0a4e40f82a2509d8bd993e54b0853c3c05465d5dc30ae829"} Nov 01 22:15:03.154602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.154582 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerStarted","Data":"00b7d52e8465f375d4f53a52beee8444c9830cacfdf0b80c4a0549c79eeb3ed3"} Nov 01 22:15:03.158524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.158429 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" podStartSLOduration=3.158417413 podStartE2EDuration="3.158417413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.157273555 +0000 UTC m=+1856.850254042" watchObservedRunningTime="2026-11-01 22:15:03.158417413 +0000 UTC m=+1856.851397898" Nov 01 22:15:03.176551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.176439 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" podStartSLOduration=3.176427448 podStartE2EDuration="3.176427448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.174521651 +0000 UTC m=+1856.867502138" watchObservedRunningTime="2026-11-01 22:15:03.176427448 +0000 UTC m=+1856.869407929" Nov 01 22:15:03.228848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.228797 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" podStartSLOduration=3.228778468 podStartE2EDuration="3.228778468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.218683895 +0000 UTC m=+1856.911664377" watchObservedRunningTime="2026-11-01 22:15:03.228778468 +0000 UTC m=+1856.921758956" Nov 01 22:15:03.243916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.241578 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" podStartSLOduration=3.241560808 podStartE2EDuration="3.241560808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.237612449 +0000 UTC m=+1856.930592932" watchObservedRunningTime="2026-11-01 22:15:03.241560808 +0000 UTC m=+1856.934541295" Nov 01 22:15:03.262634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.262577 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" podStartSLOduration=3.262560268 podStartE2EDuration="3.262560268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.26162408 +0000 UTC m=+1856.954604578" watchObservedRunningTime="2026-11-01 22:15:03.262560268 +0000 UTC m=+1856.955540755" Nov 01 22:15:03.302874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.302822 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" podStartSLOduration=3.30280459 podStartE2EDuration="3.30280459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.300246565 +0000 UTC m=+1856.993227052" watchObservedRunningTime="2026-11-01 22:15:03.30280459 +0000 UTC m=+1856.995785079" Nov 01 22:15:03.325634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.325591 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" podStartSLOduration=3.3255784 podStartE2EDuration="3.3255784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.319847684 +0000 UTC m=+1857.012828173" watchObservedRunningTime="2026-11-01 22:15:03.3255784 +0000 UTC m=+1857.018558885" Nov 01 22:15:03.342191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.342138 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" podStartSLOduration=3.342123092 podStartE2EDuration="3.342123092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.339506894 +0000 UTC m=+1857.032487380" watchObservedRunningTime="2026-11-01 22:15:03.342123092 +0000 UTC m=+1857.035103577" Nov 01 22:15:03.730973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:03.730943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:04.170392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.170351 5036 generic.go:356] "Generic (PLEG): container finished" podID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerID="1fe289902fb03904f42cda0ef6dd93b5a97a4f794ffc805724d25ea328a4e15b" exitCode=60 Nov 01 22:15:04.170919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.170424 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerDied","Data":"1fe289902fb03904f42cda0ef6dd93b5a97a4f794ffc805724d25ea328a4e15b"} Nov 01 22:15:04.170919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.170764 5036 scope.go:122] "RemoveContainer" containerID="1fe289902fb03904f42cda0ef6dd93b5a97a4f794ffc805724d25ea328a4e15b" Nov 01 22:15:04.173126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.173103 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerID="0ac048d1eaf4c4c350a4cbd27130dc8a52f60c4c9649e396583a128a1fdb9b30" exitCode=60 Nov 01 22:15:04.173216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.173155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerDied","Data":"0ac048d1eaf4c4c350a4cbd27130dc8a52f60c4c9649e396583a128a1fdb9b30"} Nov 01 22:15:04.173560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.173533 5036 scope.go:122] "RemoveContainer" containerID="0ac048d1eaf4c4c350a4cbd27130dc8a52f60c4c9649e396583a128a1fdb9b30" Nov 01 22:15:04.175113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.175076 5036 generic.go:356] "Generic (PLEG): container finished" podID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerID="7b962287852485121408b8ccf63936492b87b2de7028a8c17c591e5600c5cff0" exitCode=60 Nov 01 22:15:04.175197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.175168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerDied","Data":"7b962287852485121408b8ccf63936492b87b2de7028a8c17c591e5600c5cff0"} Nov 01 22:15:04.175404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.175385 5036 scope.go:122] "RemoveContainer" containerID="7b962287852485121408b8ccf63936492b87b2de7028a8c17c591e5600c5cff0" Nov 01 22:15:04.183144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.183122 5036 generic.go:356] "Generic (PLEG): container finished" podID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerID="6a5aaa8b33091c2cb8c5ffe697139f9329e1b824581f10cc918074efe0a51b5e" exitCode=60 Nov 01 22:15:04.183243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.183209 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerDied","Data":"6a5aaa8b33091c2cb8c5ffe697139f9329e1b824581f10cc918074efe0a51b5e"} Nov 01 22:15:04.183431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.183416 5036 scope.go:122] "RemoveContainer" containerID="6a5aaa8b33091c2cb8c5ffe697139f9329e1b824581f10cc918074efe0a51b5e" Nov 01 22:15:04.185881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.185863 5036 generic.go:356] "Generic (PLEG): container finished" podID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerID="38d9685860a9f28699a24cb4fbcd02a17ed306a69214ea54d776d9580efca58f" exitCode=60 Nov 01 22:15:04.185945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.185933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerDied","Data":"38d9685860a9f28699a24cb4fbcd02a17ed306a69214ea54d776d9580efca58f"} Nov 01 22:15:04.186182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.186165 5036 scope.go:122] "RemoveContainer" containerID="38d9685860a9f28699a24cb4fbcd02a17ed306a69214ea54d776d9580efca58f" Nov 01 22:15:04.189364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.189345 5036 generic.go:356] "Generic (PLEG): container finished" podID="699f6c9b-412f-4987-85e9-3308e01ae622" containerID="d32962f9097a8c8fc06806ef670eb9d42987740d592f3c1a60dd4dd03adefe56" exitCode=60 Nov 01 22:15:04.189628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.189607 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerDied","Data":"d32962f9097a8c8fc06806ef670eb9d42987740d592f3c1a60dd4dd03adefe56"} Nov 01 22:15:04.189777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.189761 5036 scope.go:122] "RemoveContainer" containerID="d32962f9097a8c8fc06806ef670eb9d42987740d592f3c1a60dd4dd03adefe56" Nov 01 22:15:04.191287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.191263 5036 generic.go:356] "Generic (PLEG): container finished" podID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerID="234e4a42c5b39b347efcadbd9c539565102f569781cf5896569f469ec80e4c88" exitCode=60 Nov 01 22:15:04.191371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.191344 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerDied","Data":"234e4a42c5b39b347efcadbd9c539565102f569781cf5896569f469ec80e4c88"} Nov 01 22:15:04.191777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.191754 5036 scope.go:122] "RemoveContainer" containerID="234e4a42c5b39b347efcadbd9c539565102f569781cf5896569f469ec80e4c88" Nov 01 22:15:04.193324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.193304 5036 generic.go:356] "Generic (PLEG): container finished" podID="24488105-d393-4f74-bf75-17d186b794a2" containerID="3f2881a350bd48eb6801969e617f8b44c78be56d93c5146de41fc9d56be9a84e" exitCode=60 Nov 01 22:15:04.193425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.193386 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerDied","Data":"3f2881a350bd48eb6801969e617f8b44c78be56d93c5146de41fc9d56be9a84e"} Nov 01 22:15:04.194052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.193610 5036 scope.go:122] "RemoveContainer" containerID="3f2881a350bd48eb6801969e617f8b44c78be56d93c5146de41fc9d56be9a84e" Nov 01 22:15:04.195185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.195167 5036 generic.go:356] "Generic (PLEG): container finished" podID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerID="e5b4042fa7422a8ebd679c3692a929e8bd5c4c339940288fb4ee179c816329bc" exitCode=60 Nov 01 22:15:04.195272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.195248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerDied","Data":"e5b4042fa7422a8ebd679c3692a929e8bd5c4c339940288fb4ee179c816329bc"} Nov 01 22:15:04.195733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.195436 5036 scope.go:122] "RemoveContainer" containerID="e5b4042fa7422a8ebd679c3692a929e8bd5c4c339940288fb4ee179c816329bc" Nov 01 22:15:04.197173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.197154 5036 generic.go:356] "Generic (PLEG): container finished" podID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerID="69d276e634564839675997ae36ac70612297ae9fd092b3be7a40cb48267aebaf" exitCode=60 Nov 01 22:15:04.197390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.197364 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerDied","Data":"69d276e634564839675997ae36ac70612297ae9fd092b3be7a40cb48267aebaf"} Nov 01 22:15:04.198573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.198415 5036 scope.go:122] "RemoveContainer" containerID="69d276e634564839675997ae36ac70612297ae9fd092b3be7a40cb48267aebaf" Nov 01 22:15:04.200057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.200036 5036 generic.go:356] "Generic (PLEG): container finished" podID="380a1985-8962-4447-9d6a-4a399646e5eb" containerID="c3e0bb874ba7206017470e0016f6d05a0302361bae2237a16780b9a1533496b1" exitCode=60 Nov 01 22:15:04.200159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.200138 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" event={"ID":"380a1985-8962-4447-9d6a-4a399646e5eb","Type":"ContainerDied","Data":"c3e0bb874ba7206017470e0016f6d05a0302361bae2237a16780b9a1533496b1"} Nov 01 22:15:04.200372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.200355 5036 scope.go:122] "RemoveContainer" containerID="c3e0bb874ba7206017470e0016f6d05a0302361bae2237a16780b9a1533496b1" Nov 01 22:15:04.201951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.201931 5036 generic.go:356] "Generic (PLEG): container finished" podID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerID="00b7d52e8465f375d4f53a52beee8444c9830cacfdf0b80c4a0549c79eeb3ed3" exitCode=60 Nov 01 22:15:04.202041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.201974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerDied","Data":"00b7d52e8465f375d4f53a52beee8444c9830cacfdf0b80c4a0549c79eeb3ed3"} Nov 01 22:15:04.202269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.202250 5036 scope.go:122] "RemoveContainer" containerID="00b7d52e8465f375d4f53a52beee8444c9830cacfdf0b80c4a0549c79eeb3ed3" Nov 01 22:15:04.527043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:04.526894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:05.213691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.213661 5036 generic.go:356] "Generic (PLEG): container finished" podID="86fc72a8-060e-4573-b43c-50c40521abf0" containerID="e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd" exitCode=60 Nov 01 22:15:05.214113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.213742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerDied","Data":"e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd"} Nov 01 22:15:05.214113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.213829 5036 scope.go:122] "RemoveContainer" containerID="0947b4a75d6f3cac283019a69649aaca7d05bbda7fc5c35babd6e54bb3c285a5" Nov 01 22:15:05.214247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.214113 5036 scope.go:122] "RemoveContainer" containerID="e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd" Nov 01 22:15:05.214555 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:05.214530 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9_odh-ai-gateway-infra(86fc72a8-060e-4573-b43c-50c40521abf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" Nov 01 22:15:05.216401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.216373 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerStarted","Data":"087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34"} Nov 01 22:15:05.218715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.218679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerStarted","Data":"cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3"} Nov 01 22:15:05.221093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.221071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerStarted","Data":"9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5"} Nov 01 22:15:05.223312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.223288 5036 generic.go:356] "Generic (PLEG): container finished" podID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerID="44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf" exitCode=60 Nov 01 22:15:05.223389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.223365 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerDied","Data":"44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf"} Nov 01 22:15:05.223642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.223626 5036 scope.go:122] "RemoveContainer" containerID="44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf" Nov 01 22:15:05.223913 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:05.223886 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx_odh-ai-gateway-infra(14f13ed8-24f7-493b-8c7f-a8d632156d69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" Nov 01 22:15:05.225433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.225414 5036 generic.go:356] "Generic (PLEG): container finished" podID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerID="00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba" exitCode=60 Nov 01 22:15:05.225539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.225491 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerDied","Data":"00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba"} Nov 01 22:15:05.225713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.225698 5036 scope.go:122] "RemoveContainer" containerID="00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba" Nov 01 22:15:05.225926 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:05.225910 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2_odh-ai-gateway-infra(c1448ce4-6179-435e-b9fc-55a5e6b5a7c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" Nov 01 22:15:05.257077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.256990 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" podStartSLOduration=5.256974123 podStartE2EDuration="5.256974123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:05.251599368 +0000 UTC m=+1858.944579867" watchObservedRunningTime="2026-11-01 22:15:05.256974123 +0000 UTC m=+1858.949954610" Nov 01 22:15:05.342505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.342382 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:05.460768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.460743 5036 scope.go:122] "RemoveContainer" containerID="2a89e6e8f5d7f469a2dc75a685ae4c2e768409d38db51c3e9d216bceb2431b65" Nov 01 22:15:05.484431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.483124 5036 scope.go:122] "RemoveContainer" containerID="be843eb6b7874ee5b6ad6478835b0cd6c1d4d2903c74f5982030fe561a057481" Nov 01 22:15:05.665430 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:05.665397 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24488105_d393_4f74_bf75_17d186b794a2.slice/crio-9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:15:05.675790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.675754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:05.799884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.799848 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:05.937141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.937110 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64"] Nov 01 22:15:05.942718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.942696 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:05.948202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.948177 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:05.952062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:05.952039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64"] Nov 01 22:15:06.041791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.041755 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.042072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.042044 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.042170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.042151 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdm4c\" (UniqueName: \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.144421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.144389 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.145390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.144491 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdm4c\" (UniqueName: \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.145390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.144744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.145542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.145407 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.147293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.147271 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.159135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.159110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdm4c\" (UniqueName: \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\") pod \"maas-api-revoke-sub-b16fe198-b9s64\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.235341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.235302 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:06.240189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.240169 5036 scope.go:122] "RemoveContainer" containerID="44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf" Nov 01 22:15:06.240640 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.240614 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx_odh-ai-gateway-infra(14f13ed8-24f7-493b-8c7f-a8d632156d69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" Nov 01 22:15:06.271971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.271529 5036 scope.go:122] "RemoveContainer" containerID="2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1" Nov 01 22:15:06.271971 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.272027 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg_odh-ai-gateway-infra(a7b475c9-75d6-4ce0-9e01-3cf85742b1bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" Nov 01 22:15:06.274946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.270893 5036 generic.go:356] "Generic (PLEG): container finished" podID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerID="2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1" exitCode=60 Nov 01 22:15:06.274946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.272601 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerDied","Data":"2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1"} Nov 01 22:15:06.274946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.272681 5036 scope.go:122] "RemoveContainer" containerID="e5b4042fa7422a8ebd679c3692a929e8bd5c4c339940288fb4ee179c816329bc" Nov 01 22:15:06.280505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.280473 5036 generic.go:356] "Generic (PLEG): container finished" podID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerID="780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58" exitCode=60 Nov 01 22:15:06.280613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.280565 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerDied","Data":"780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58"} Nov 01 22:15:06.281988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.281961 5036 scope.go:122] "RemoveContainer" containerID="780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58" Nov 01 22:15:06.282811 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.282781 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx_odh-ai-gateway-infra(7be5bd1f-7a22-44ec-92f8-f156941b97c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" Nov 01 22:15:06.289080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.289033 5036 scope.go:122] "RemoveContainer" containerID="00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba" Nov 01 22:15:06.290760 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.290718 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2_odh-ai-gateway-infra(c1448ce4-6179-435e-b9fc-55a5e6b5a7c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" Nov 01 22:15:06.304805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.304682 5036 generic.go:356] "Generic (PLEG): container finished" podID="380a1985-8962-4447-9d6a-4a399646e5eb" containerID="fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8" exitCode=60 Nov 01 22:15:06.304920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.304802 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" event={"ID":"380a1985-8962-4447-9d6a-4a399646e5eb","Type":"ContainerDied","Data":"fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8"} Nov 01 22:15:06.306049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.305237 5036 scope.go:122] "RemoveContainer" containerID="fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8" Nov 01 22:15:06.306049 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.305701 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689_odh-ai-gateway-infra(380a1985-8962-4447-9d6a-4a399646e5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" Nov 01 22:15:06.326099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.326054 5036 generic.go:356] "Generic (PLEG): container finished" podID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerID="604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26" exitCode=60 Nov 01 22:15:06.326425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.326123 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerDied","Data":"604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26"} Nov 01 22:15:06.326790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.326643 5036 scope.go:122] "RemoveContainer" containerID="604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26" Nov 01 22:15:06.327213 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.327176 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm_odh-ai-gateway-infra(a1d159d2-492c-4e80-b0d4-2bb118563a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" Nov 01 22:15:06.330270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.330246 5036 generic.go:356] "Generic (PLEG): container finished" podID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerID="9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276" exitCode=60 Nov 01 22:15:06.330367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.330325 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerDied","Data":"9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276"} Nov 01 22:15:06.330796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.330776 5036 scope.go:122] "RemoveContainer" containerID="9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276" Nov 01 22:15:06.331234 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.331213 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg_odh-ai-gateway-infra(03a3f22e-9730-434a-828b-db0cc6a82ef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" Nov 01 22:15:06.332682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.332662 5036 scope.go:122] "RemoveContainer" containerID="e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd" Nov 01 22:15:06.333057 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.333035 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9_odh-ai-gateway-infra(86fc72a8-060e-4573-b43c-50c40521abf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" Nov 01 22:15:06.334832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.334752 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerID="fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756" exitCode=60 Nov 01 22:15:06.334928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.334829 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerDied","Data":"fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756"} Nov 01 22:15:06.335200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.335169 5036 scope.go:122] "RemoveContainer" containerID="fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756" Nov 01 22:15:06.335677 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.335652 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7_odh-ai-gateway-infra(0b9618b6-869f-457f-ba32-58fde8fcad69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" Nov 01 22:15:06.338739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.338690 5036 generic.go:356] "Generic (PLEG): container finished" podID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerID="ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb" exitCode=60 Nov 01 22:15:06.338833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.338822 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerDied","Data":"ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb"} Nov 01 22:15:06.339305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.339284 5036 scope.go:122] "RemoveContainer" containerID="ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb" Nov 01 22:15:06.339767 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.339710 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf_odh-ai-gateway-infra(75baa909-f44b-4da4-9756-0dc10f2ad0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" Nov 01 22:15:06.365070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.364942 5036 generic.go:356] "Generic (PLEG): container finished" podID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerID="5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb" exitCode=60 Nov 01 22:15:06.365070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.365062 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerDied","Data":"5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb"} Nov 01 22:15:06.365384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.365351 5036 scope.go:122] "RemoveContainer" containerID="69d276e634564839675997ae36ac70612297ae9fd092b3be7a40cb48267aebaf" Nov 01 22:15:06.366156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.366134 5036 scope.go:122] "RemoveContainer" containerID="5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb" Nov 01 22:15:06.366531 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.366508 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7_odh-ai-gateway-infra(07f6a84f-1958-4d61-bfde-b61102cdd8d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" Nov 01 22:15:06.368540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.368509 5036 generic.go:356] "Generic (PLEG): container finished" podID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerID="59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673" exitCode=60 Nov 01 22:15:06.368540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.368527 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerDied","Data":"59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673"} Nov 01 22:15:06.368949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.368844 5036 scope.go:122] "RemoveContainer" containerID="59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673" Nov 01 22:15:06.369322 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.369299 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv_odh-ai-gateway-infra(27fb36f2-9ac7-4704-930d-0e25aa26879a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" Nov 01 22:15:06.396169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.396080 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:06.397059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.397029 5036 generic.go:356] "Generic (PLEG): container finished" podID="699f6c9b-412f-4987-85e9-3308e01ae622" containerID="087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34" exitCode=60 Nov 01 22:15:06.397893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.397369 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerDied","Data":"087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34"} Nov 01 22:15:06.398455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.398417 5036 scope.go:122] "RemoveContainer" containerID="087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34" Nov 01 22:15:06.403658 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.400214 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t_odh-ai-gateway-infra(699f6c9b-412f-4987-85e9-3308e01ae622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" Nov 01 22:15:06.403991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.403970 5036 generic.go:356] "Generic (PLEG): container finished" podID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerID="cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3" exitCode=60 Nov 01 22:15:06.404157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.404053 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerDied","Data":"cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3"} Nov 01 22:15:06.404557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.404283 5036 scope.go:122] "RemoveContainer" containerID="cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3" Nov 01 22:15:06.404737 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.404622 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc_odh-ai-gateway-infra(c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" Nov 01 22:15:06.406695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.406676 5036 generic.go:356] "Generic (PLEG): container finished" podID="24488105-d393-4f74-bf75-17d186b794a2" containerID="9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5" exitCode=60 Nov 01 22:15:06.406785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.406755 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerDied","Data":"9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5"} Nov 01 22:15:06.407216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.407199 5036 scope.go:122] "RemoveContainer" containerID="9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5" Nov 01 22:15:06.407741 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:06.407715 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm_odh-ai-gateway-infra(24488105-d393-4f74-bf75-17d186b794a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" podUID="24488105-d393-4f74-bf75-17d186b794a2" Nov 01 22:15:06.409973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.409953 5036 scope.go:122] "RemoveContainer" containerID="c3e0bb874ba7206017470e0016f6d05a0302361bae2237a16780b9a1533496b1" Nov 01 22:15:06.431622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.431598 5036 scope.go:122] "RemoveContainer" containerID="00b7d52e8465f375d4f53a52beee8444c9830cacfdf0b80c4a0549c79eeb3ed3" Nov 01 22:15:06.450780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.450753 5036 scope.go:122] "RemoveContainer" containerID="1fe289902fb03904f42cda0ef6dd93b5a97a4f794ffc805724d25ea328a4e15b" Nov 01 22:15:06.467993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.467974 5036 scope.go:122] "RemoveContainer" containerID="0ac048d1eaf4c4c350a4cbd27130dc8a52f60c4c9649e396583a128a1fdb9b30" Nov 01 22:15:06.501000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.500248 5036 scope.go:122] "RemoveContainer" containerID="7b962287852485121408b8ccf63936492b87b2de7028a8c17c591e5600c5cff0" Nov 01 22:15:06.541325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.540873 5036 scope.go:122] "RemoveContainer" containerID="6a5aaa8b33091c2cb8c5ffe697139f9329e1b824581f10cc918074efe0a51b5e" Nov 01 22:15:06.561343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.561321 5036 scope.go:122] "RemoveContainer" containerID="38d9685860a9f28699a24cb4fbcd02a17ed306a69214ea54d776d9580efca58f" Nov 01 22:15:06.599654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.598984 5036 scope.go:122] "RemoveContainer" containerID="d32962f9097a8c8fc06806ef670eb9d42987740d592f3c1a60dd4dd03adefe56" Nov 01 22:15:06.630766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.630742 5036 scope.go:122] "RemoveContainer" containerID="234e4a42c5b39b347efcadbd9c539565102f569781cf5896569f469ec80e4c88" Nov 01 22:15:06.652402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.652372 5036 scope.go:122] "RemoveContainer" containerID="3f2881a350bd48eb6801969e617f8b44c78be56d93c5146de41fc9d56be9a84e" Nov 01 22:15:06.708247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:06.708225 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64"] Nov 01 22:15:06.708580 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:06.708550 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca9fcf4_76e0_491b_b852_b07cd569facb.slice/crio-dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787 WatchSource:0}: Error finding container dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787: Status 404 returned error can't find the container with id dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787 Nov 01 22:15:07.150845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.150803 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:07.419229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.419195 5036 scope.go:122] "RemoveContainer" containerID="fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756" Nov 01 22:15:07.419758 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.419561 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7_odh-ai-gateway-infra(0b9618b6-869f-457f-ba32-58fde8fcad69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" Nov 01 22:15:07.421339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.421307 5036 scope.go:122] "RemoveContainer" containerID="ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb" Nov 01 22:15:07.421822 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.421790 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf_odh-ai-gateway-infra(75baa909-f44b-4da4-9756-0dc10f2ad0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" Nov 01 22:15:07.423330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.423309 5036 scope.go:122] "RemoveContainer" containerID="5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb" Nov 01 22:15:07.423739 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.423711 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7_odh-ai-gateway-infra(07f6a84f-1958-4d61-bfde-b61102cdd8d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" Nov 01 22:15:07.425668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.425649 5036 scope.go:122] "RemoveContainer" containerID="59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673" Nov 01 22:15:07.425984 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.425962 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv_odh-ai-gateway-infra(27fb36f2-9ac7-4704-930d-0e25aa26879a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" Nov 01 22:15:07.427421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.427403 5036 scope.go:122] "RemoveContainer" containerID="087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34" Nov 01 22:15:07.427709 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.427684 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t_odh-ai-gateway-infra(699f6c9b-412f-4987-85e9-3308e01ae622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" Nov 01 22:15:07.429142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.429114 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" event={"ID":"0ca9fcf4-76e0-491b-b852-b07cd569facb","Type":"ContainerStarted","Data":"cededc50a3dc0954145707995ec3ec727685b31ce572f613846126ecc2029f41"} Nov 01 22:15:07.429252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.429170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" event={"ID":"0ca9fcf4-76e0-491b-b852-b07cd569facb","Type":"ContainerStarted","Data":"dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787"} Nov 01 22:15:07.431161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.431142 5036 scope.go:122] "RemoveContainer" containerID="cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3" Nov 01 22:15:07.431391 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.431369 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc_odh-ai-gateway-infra(c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" Nov 01 22:15:07.432986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.432969 5036 scope.go:122] "RemoveContainer" containerID="9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5" Nov 01 22:15:07.433204 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.433181 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm_odh-ai-gateway-infra(24488105-d393-4f74-bf75-17d186b794a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" podUID="24488105-d393-4f74-bf75-17d186b794a2" Nov 01 22:15:07.434969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.434950 5036 scope.go:122] "RemoveContainer" containerID="2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1" Nov 01 22:15:07.435214 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.435195 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg_odh-ai-gateway-infra(a7b475c9-75d6-4ce0-9e01-3cf85742b1bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" Nov 01 22:15:07.436749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.436731 5036 scope.go:122] "RemoveContainer" containerID="780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58" Nov 01 22:15:07.436999 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.436980 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx_odh-ai-gateway-infra(7be5bd1f-7a22-44ec-92f8-f156941b97c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" Nov 01 22:15:07.439053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.439037 5036 scope.go:122] "RemoveContainer" containerID="fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8" Nov 01 22:15:07.439429 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.439404 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689_odh-ai-gateway-infra(380a1985-8962-4447-9d6a-4a399646e5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" Nov 01 22:15:07.440841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.440822 5036 scope.go:122] "RemoveContainer" containerID="604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26" Nov 01 22:15:07.441096 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.441070 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm_odh-ai-gateway-infra(a1d159d2-492c-4e80-b0d4-2bb118563a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" Nov 01 22:15:07.442819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.442803 5036 scope.go:122] "RemoveContainer" containerID="9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276" Nov 01 22:15:07.443040 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:07.443022 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg_odh-ai-gateway-infra(03a3f22e-9730-434a-828b-db0cc6a82ef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" Nov 01 22:15:07.482270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.482223 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" podStartSLOduration=2.482207567 podStartE2EDuration="2.482207567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:07.476131496 +0000 UTC m=+1861.169111995" watchObservedRunningTime="2026-11-01 22:15:07.482207567 +0000 UTC m=+1861.175188056" Nov 01 22:15:07.961660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:07.961556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:08.455007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:08.454973 5036 generic.go:356] "Generic (PLEG): container finished" podID="0ca9fcf4-76e0-491b-b852-b07cd569facb" containerID="cededc50a3dc0954145707995ec3ec727685b31ce572f613846126ecc2029f41" exitCode=0 Nov 01 22:15:08.455413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:08.455054 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" event={"ID":"0ca9fcf4-76e0-491b-b852-b07cd569facb","Type":"ContainerDied","Data":"cededc50a3dc0954145707995ec3ec727685b31ce572f613846126ecc2029f41"} Nov 01 22:15:08.632372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:08.632328 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:09.495739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.495687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:09.753418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.753386 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:09.863175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.863140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\") pod \"0ca9fcf4-76e0-491b-b852-b07cd569facb\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " Nov 01 22:15:09.863350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.863211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\") pod \"0ca9fcf4-76e0-491b-b852-b07cd569facb\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " Nov 01 22:15:09.863467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.863365 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\" (UniqueName: \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\") pod \"0ca9fcf4-76e0-491b-b852-b07cd569facb\" (UID: \"0ca9fcf4-76e0-491b-b852-b07cd569facb\") " Nov 01 22:15:09.863656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.863629 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca" pod "0ca9fcf4-76e0-491b-b852-b07cd569facb" (UID: "0ca9fcf4-76e0-491b-b852-b07cd569facb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:09.864415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.864394 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:09.866281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.866247 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c" pod "0ca9fcf4-76e0-491b-b852-b07cd569facb" (UID: "0ca9fcf4-76e0-491b-b852-b07cd569facb"). InnerVolumeSpecName "kube-api-access-mdm4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:09.866381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.866321 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca" pod "0ca9fcf4-76e0-491b-b852-b07cd569facb" (UID: "0ca9fcf4-76e0-491b-b852-b07cd569facb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:09.966026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.965994 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca9fcf4-76e0-491b-b852-b07cd569facb-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:09.966026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:09.966023 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdm4c\" (UniqueName: \"kubernetes.io/projected/0ca9fcf4-76e0-491b-b852-b07cd569facb-kube-api-access-mdm4c\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:10.038266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.038236 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:10.170068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.170026 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:10.300841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.300714 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:10.478913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.478888 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" Nov 01 22:15:10.479098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.478925 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64" event={"ID":"0ca9fcf4-76e0-491b-b852-b07cd569facb","Type":"ContainerDied","Data":"dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787"} Nov 01 22:15:10.479098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:10.478970 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9b9960e5602d8afab22ac4c0f4d28d36990b397769b2bbce55a38bde63a787" Nov 01 22:15:11.045507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:11.045426 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:11.925035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:11.924992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:12.641419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:12.641380 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:13.434010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:13.433958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:14.339123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:14.339091 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:15.251815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:15.251772 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:15.574852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:15.574813 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:15:15.634416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:15.634364 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:15.882900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:15.882818 5036 scope.go:122] "RemoveContainer" containerID="84292c8fef23d6ab3563b7d3c1838250a44ef6200d0ebc86072488b9634b7edf" Nov 01 22:15:15.898976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:15.898946 5036 scope.go:122] "RemoveContainer" containerID="59b765023d34d629e52e5231e4fc7950405fc6bbad911ec63d8c8e387af7d7fd" Nov 01 22:15:16.138599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.138554 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6"] Nov 01 22:15:16.143814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.142292 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0ca9fcf4-76e0-491b-b852-b07cd569facb" containerName="revoke-keys" Nov 01 22:15:16.143999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.143979 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca9fcf4-76e0-491b-b852-b07cd569facb" containerName="revoke-keys" Nov 01 22:15:16.144666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.144645 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca9fcf4-76e0-491b-b852-b07cd569facb" containerName="revoke-keys" Nov 01 22:15:16.211176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.211141 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.213348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.213313 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6"] Nov 01 22:15:16.216332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.216306 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:16.317245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.311704 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.319507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.319416 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mcs\" (UniqueName: \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.319663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.319646 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.421162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.421081 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.421349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.421279 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.421349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.421339 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mcs\" (UniqueName: \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.422087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.422056 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.424161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.424130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.434034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.434008 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mcs\" (UniqueName: \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\") pod \"maas-api-revoke-sub-ad872dc3-mjxm6\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.436532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.436454 5036 scope.go:122] "RemoveContainer" containerID="604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26" Nov 01 22:15:16.436669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.436632 5036 scope.go:122] "RemoveContainer" containerID="44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf" Nov 01 22:15:16.437109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.437045 5036 scope.go:122] "RemoveContainer" containerID="2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1" Nov 01 22:15:16.437399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.437348 5036 scope.go:122] "RemoveContainer" containerID="9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5" Nov 01 22:15:16.496532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.496498 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:16.557373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.557042 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:16.905008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:16.904979 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6"] Nov 01 22:15:16.979966 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:16.979881 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dc55de_462e_44a5_a008_750cc7ff28f7.slice/crio-20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4 WatchSource:0}: Error finding container 20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4: Status 404 returned error can't find the container with id 20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4 Nov 01 22:15:17.460478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.436788 5036 scope.go:122] "RemoveContainer" containerID="59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673" Nov 01 22:15:17.460478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.437940 5036 scope.go:122] "RemoveContainer" containerID="cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3" Nov 01 22:15:17.460478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.442224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:17.460478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.456287 5036 scope.go:122] "RemoveContainer" containerID="9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276" Nov 01 22:15:17.460478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.457154 5036 scope.go:122] "RemoveContainer" containerID="e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd" Nov 01 22:15:17.710118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.710083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:17.867530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.867314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerStarted","Data":"fa40d2df687c628e82a1f47ec9143d32b3fe8733364e02ff88adddd3e5461710"} Nov 01 22:15:17.896057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.895981 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" podStartSLOduration=17.895956506 podStartE2EDuration="17.895956506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:17.887209848 +0000 UTC m=+1871.580190335" watchObservedRunningTime="2026-11-01 22:15:17.895956506 +0000 UTC m=+1871.588936993" Nov 01 22:15:17.903758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.903727 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerStarted","Data":"7920114e4d95bf08af5b99bf52024f32416cd44e49c98673664089e313b0f391"} Nov 01 22:15:17.906438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.906343 5036 scope.go:122] "RemoveContainer" containerID="7920114e4d95bf08af5b99bf52024f32416cd44e49c98673664089e313b0f391" Nov 01 22:15:17.907395 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:17.907372 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm_odh-ai-gateway-infra(a1d159d2-492c-4e80-b0d4-2bb118563a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" Nov 01 22:15:17.908479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:17.908414 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" event={"ID":"52dc55de-462e-44a5-a008-750cc7ff28f7","Type":"ContainerStarted","Data":"20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4"} Nov 01 22:15:18.008672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.008644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:18.457467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.454812 5036 scope.go:122] "RemoveContainer" containerID="780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58" Nov 01 22:15:18.457467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.456186 5036 scope.go:122] "RemoveContainer" containerID="00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba" Nov 01 22:15:18.476522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.458551 5036 scope.go:122] "RemoveContainer" containerID="5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb" Nov 01 22:15:18.484819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.484766 5036 scope.go:122] "RemoveContainer" containerID="fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756" Nov 01 22:15:18.628560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.622271 5036 scope.go:122] "RemoveContainer" containerID="087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34" Nov 01 22:15:18.849676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.848665 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:18.914389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.914307 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:18.932480 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:18.932425 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:15:18.993175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.993083 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:18.999763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.999728 5036 generic.go:356] "Generic (PLEG): container finished" podID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerID="fa40d2df687c628e82a1f47ec9143d32b3fe8733364e02ff88adddd3e5461710" exitCode=60 Nov 01 22:15:18.999902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.999881 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerDied","Data":"fa40d2df687c628e82a1f47ec9143d32b3fe8733364e02ff88adddd3e5461710"} Nov 01 22:15:18.999968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:18.999938 5036 scope.go:122] "RemoveContainer" containerID="44f6a95ee549a8d6bafb3f29a96fbe9872879835edb04a7279f3231f073433cf" Nov 01 22:15:19.008022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.008001 5036 generic.go:356] "Generic (PLEG): container finished" podID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerID="5de192c7f764e7b008604521e80101d976331c5f5897adb2113bc4ad06849581" exitCode=60 Nov 01 22:15:19.008350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.008160 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerDied","Data":"5de192c7f764e7b008604521e80101d976331c5f5897adb2113bc4ad06849581"} Nov 01 22:15:19.009097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.009081 5036 scope.go:122] "RemoveContainer" containerID="5de192c7f764e7b008604521e80101d976331c5f5897adb2113bc4ad06849581" Nov 01 22:15:19.009794 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:19.009771 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg_odh-ai-gateway-infra(a7b475c9-75d6-4ce0-9e01-3cf85742b1bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" Nov 01 22:15:19.012751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.012728 5036 generic.go:356] "Generic (PLEG): container finished" podID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerID="7920114e4d95bf08af5b99bf52024f32416cd44e49c98673664089e313b0f391" exitCode=60 Nov 01 22:15:19.012867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.012844 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerDied","Data":"7920114e4d95bf08af5b99bf52024f32416cd44e49c98673664089e313b0f391"} Nov 01 22:15:19.016958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.016933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerStarted","Data":"cd4911b4d83b76ae68f01115c87a9eee708a60844e7c302f059d1ef606af090e"} Nov 01 22:15:19.026052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.026027 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerStarted","Data":"4069cfdd9ff3770941e38ab06e3deaa9ad5c4e365f3d5410a6acb3ccb0b5e248"} Nov 01 22:15:19.032521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.032386 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerStarted","Data":"166c764ff236360aad1c580dc1a7f0218f7b743a41addd454cbf1f848cfb2e0d"} Nov 01 22:15:19.036369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.036346 5036 generic.go:356] "Generic (PLEG): container finished" podID="52dc55de-462e-44a5-a008-750cc7ff28f7" containerID="fe903b7f2c07cf1c935135ed0a8da50a4739d7742b2ffea20f0bc72cf3852908" exitCode=0 Nov 01 22:15:19.036888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.036617 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" event={"ID":"52dc55de-462e-44a5-a008-750cc7ff28f7","Type":"ContainerDied","Data":"fe903b7f2c07cf1c935135ed0a8da50a4739d7742b2ffea20f0bc72cf3852908"} Nov 01 22:15:19.038700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.038682 5036 scope.go:122] "RemoveContainer" containerID="2766e8c0334ecf37d64518b14abc281c93f087b0bcb7c3c805feccd72ddd2fe1" Nov 01 22:15:19.046622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.046603 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerStarted","Data":"b35d88f053d012138621c544f41848d271223f7dbb7bf351a8d150f5e932e51e"} Nov 01 22:15:19.051045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.051021 5036 generic.go:356] "Generic (PLEG): container finished" podID="24488105-d393-4f74-bf75-17d186b794a2" containerID="07c30f5af926cab08303688873ccb7c6c1b22a8f78d69fd98478fa444ff2fdcd" exitCode=60 Nov 01 22:15:19.051153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.051136 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerDied","Data":"07c30f5af926cab08303688873ccb7c6c1b22a8f78d69fd98478fa444ff2fdcd"} Nov 01 22:15:19.051532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.051501 5036 scope.go:122] "RemoveContainer" containerID="07c30f5af926cab08303688873ccb7c6c1b22a8f78d69fd98478fa444ff2fdcd" Nov 01 22:15:19.051960 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:19.051937 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm_odh-ai-gateway-infra(24488105-d393-4f74-bf75-17d186b794a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" podUID="24488105-d393-4f74-bf75-17d186b794a2" Nov 01 22:15:19.061015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.060997 5036 scope.go:122] "RemoveContainer" containerID="604c5ebcddb57bfc597838a7e222eb3f93006f2b366f5be46857d993d7e1ed26" Nov 01 22:15:19.076334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.076306 5036 scope.go:122] "RemoveContainer" containerID="9cbb25f1cfe8b3866c49c70e1a5079c23cd6c864a8023290a0ab88fbca4ecfc5" Nov 01 22:15:19.134480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.131079 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" podStartSLOduration=19.131066159 podStartE2EDuration="19.131066159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:19.128695964 +0000 UTC m=+1872.821676450" watchObservedRunningTime="2026-11-01 22:15:19.131066159 +0000 UTC m=+1872.824046702" Nov 01 22:15:19.372407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.372123 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:19.430739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.430706 5036 scope.go:122] "RemoveContainer" containerID="ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb" Nov 01 22:15:19.432267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.432245 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\" (UniqueName: \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\") pod \"14f13ed8-24f7-493b-8c7f-a8d632156d69\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " Nov 01 22:15:19.432399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.432310 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\") pod \"14f13ed8-24f7-493b-8c7f-a8d632156d69\" (UID: \"14f13ed8-24f7-493b-8c7f-a8d632156d69\") " Nov 01 22:15:19.433832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.433802 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca" pod "14f13ed8-24f7-493b-8c7f-a8d632156d69" (UID: "14f13ed8-24f7-493b-8c7f-a8d632156d69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:19.493159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.492704 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv" pod "14f13ed8-24f7-493b-8c7f-a8d632156d69" (UID: "14f13ed8-24f7-493b-8c7f-a8d632156d69"). InnerVolumeSpecName "kube-api-access-gppmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:19.536143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.536112 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gppmv\" (UniqueName: \"kubernetes.io/projected/14f13ed8-24f7-493b-8c7f-a8d632156d69-kube-api-access-gppmv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.536236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.536148 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f13ed8-24f7-493b-8c7f-a8d632156d69-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.659934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.659907 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:19.741170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.741133 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\" (UniqueName: \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\") pod \"a1d159d2-492c-4e80-b0d4-2bb118563a87\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " Nov 01 22:15:19.741403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.741383 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\") pod \"a1d159d2-492c-4e80-b0d4-2bb118563a87\" (UID: \"a1d159d2-492c-4e80-b0d4-2bb118563a87\") " Nov 01 22:15:19.741807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.741779 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca" pod "a1d159d2-492c-4e80-b0d4-2bb118563a87" (UID: "a1d159d2-492c-4e80-b0d4-2bb118563a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:19.742250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.742224 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d159d2-492c-4e80-b0d4-2bb118563a87-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.761866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.761790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:19.779948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.779917 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb" pod "a1d159d2-492c-4e80-b0d4-2bb118563a87" (UID: "a1d159d2-492c-4e80-b0d4-2bb118563a87"). InnerVolumeSpecName "kube-api-access-8rxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:19.857748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.857702 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxqb\" (UniqueName: \"kubernetes.io/projected/a1d159d2-492c-4e80-b0d4-2bb118563a87-kube-api-access-8rxqb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.876478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.874275 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:15:19.906179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:19.906104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:20.043405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.043370 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:20.159646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.159611 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:20.368998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.335390 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:20.368998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.344837 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerStarted","Data":"444fe66f371b250908bbab9718c9dc0498c29107f4de1e375a99aa4046d5325c"} Nov 01 22:15:20.393543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.384539 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:20.396774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.396741 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerStarted","Data":"fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995"} Nov 01 22:15:20.397715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.397694 5036 scope.go:122] "RemoveContainer" containerID="fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995" Nov 01 22:15:20.398358 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:20.398320 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2_odh-ai-gateway-infra(c1448ce4-6179-435e-b9fc-55a5e6b5a7c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" Nov 01 22:15:20.399205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.399183 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" event={"ID":"a1d159d2-492c-4e80-b0d4-2bb118563a87","Type":"ContainerDied","Data":"a991f3a27ea0b4d316f130ae54bc878e8cc90260c02dd1f00664d03fc708dbf7"} Nov 01 22:15:20.399493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.399437 5036 scope.go:122] "RemoveContainer" containerID="7920114e4d95bf08af5b99bf52024f32416cd44e49c98673664089e313b0f391" Nov 01 22:15:20.399712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.399306 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm" Nov 01 22:15:20.405335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.405315 5036 generic.go:356] "Generic (PLEG): container finished" podID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerID="cd4911b4d83b76ae68f01115c87a9eee708a60844e7c302f059d1ef606af090e" exitCode=60 Nov 01 22:15:20.405560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.405541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerDied","Data":"cd4911b4d83b76ae68f01115c87a9eee708a60844e7c302f059d1ef606af090e"} Nov 01 22:15:20.413128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.412866 5036 generic.go:356] "Generic (PLEG): container finished" podID="86fc72a8-060e-4573-b43c-50c40521abf0" containerID="4069cfdd9ff3770941e38ab06e3deaa9ad5c4e365f3d5410a6acb3ccb0b5e248" exitCode=60 Nov 01 22:15:20.413128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.412950 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerDied","Data":"4069cfdd9ff3770941e38ab06e3deaa9ad5c4e365f3d5410a6acb3ccb0b5e248"} Nov 01 22:15:20.413500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.413476 5036 scope.go:122] "RemoveContainer" containerID="4069cfdd9ff3770941e38ab06e3deaa9ad5c4e365f3d5410a6acb3ccb0b5e248" Nov 01 22:15:20.430983 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:20.430948 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9_odh-ai-gateway-infra(86fc72a8-060e-4573-b43c-50c40521abf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" Nov 01 22:15:20.431268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.414203 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:20.431557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.431401 5036 scope.go:122] "RemoveContainer" containerID="9381ae98e33ced3b6cd836782f29d786f3cdc265463370b49e87d27808845276" Nov 01 22:15:20.433111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.433067 5036 scope.go:122] "RemoveContainer" containerID="fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8" Nov 01 22:15:20.450839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.450789 5036 generic.go:356] "Generic (PLEG): container finished" podID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerID="166c764ff236360aad1c580dc1a7f0218f7b743a41addd454cbf1f848cfb2e0d" exitCode=60 Nov 01 22:15:20.462011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.461864 5036 generic.go:356] "Generic (PLEG): container finished" podID="699f6c9b-412f-4987-85e9-3308e01ae622" containerID="40431933a180986ea4338421cd024f4fc93317134fed06064ae9888055d5d01a" exitCode=60 Nov 01 22:15:20.465626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.465563 5036 generic.go:356] "Generic (PLEG): container finished" podID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerID="b35d88f053d012138621c544f41848d271223f7dbb7bf351a8d150f5e932e51e" exitCode=60 Nov 01 22:15:20.486483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.481968 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" Nov 01 22:15:20.514127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.514069 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:20.514399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.514372 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerStarted","Data":"ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6"} Nov 01 22:15:20.514607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.514591 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:20.514787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.514774 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:20.514905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.514890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerStarted","Data":"a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9"} Nov 01 22:15:20.515022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.515008 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerDied","Data":"166c764ff236360aad1c580dc1a7f0218f7b743a41addd454cbf1f848cfb2e0d"} Nov 01 22:15:20.517194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.517168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerDied","Data":"40431933a180986ea4338421cd024f4fc93317134fed06064ae9888055d5d01a"} Nov 01 22:15:20.517368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.517352 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerDied","Data":"b35d88f053d012138621c544f41848d271223f7dbb7bf351a8d150f5e932e51e"} Nov 01 22:15:20.517527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.517510 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx" event={"ID":"14f13ed8-24f7-493b-8c7f-a8d632156d69","Type":"ContainerDied","Data":"a84eb30a6d9c08b1ae7a01408ccfb02d5e5adc88bd3552038325b3309d5a28a4"} Nov 01 22:15:20.519575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.516187 5036 scope.go:122] "RemoveContainer" containerID="a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9" Nov 01 22:15:20.519575 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:20.518332 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7_odh-ai-gateway-infra(07f6a84f-1958-4d61-bfde-b61102cdd8d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" Nov 01 22:15:20.519575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.518377 5036 scope.go:122] "RemoveContainer" containerID="ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6" Nov 01 22:15:20.519575 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:20.518664 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7_odh-ai-gateway-infra(0b9618b6-869f-457f-ba32-58fde8fcad69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" Nov 01 22:15:20.556731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.556632 5036 scope.go:122] "RemoveContainer" containerID="e43ef8dff1e5c4ba0d7234e814b91752b349654d74a76e3f8dd7b65c2d7c46bd" Nov 01 22:15:20.564559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.564528 5036 scope.go:122] "RemoveContainer" containerID="40431933a180986ea4338421cd024f4fc93317134fed06064ae9888055d5d01a" Nov 01 22:15:20.565382 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:20.565008 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t_odh-ai-gateway-infra(699f6c9b-412f-4987-85e9-3308e01ae622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.566426 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z"] Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567724 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567744 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567775 5036 cpu_manager.go:414] "containerMap: removing container" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567783 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567838 5036 cpu_manager.go:414] "containerMap: removing container" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567846 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567932 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.567940 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.568183 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.568199 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.568212 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.568222 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:20.568579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.568240 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:20.578559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.576943 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:20.616282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.616068 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z"] Nov 01 22:15:20.798478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.778754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:20.799548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.793621 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:20.799548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.798954 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pnq\" (UniqueName: \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:20.799548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.799110 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.005935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.929301 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.005935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.929386 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pnq\" (UniqueName: \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.005935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.929526 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.005935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:20.930331 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.006369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.006338 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:21.166467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.153403 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pnq\" (UniqueName: \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.185921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.167710 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892855-rkndm"] Nov 01 22:15:21.225105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.225026 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd730292-sn25z\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.274854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.274731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:21.337651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.337356 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:21.586039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.585137 5036 scope.go:122] "RemoveContainer" containerID="59f2d3ba2e558fc02c5469a071560e6867bfc3d5fea641a36b80d82113ac7673" Nov 01 22:15:21.649925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.649816 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:21.719718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.719690 5036 scope.go:122] "RemoveContainer" containerID="087a2adbe751d2698c8357592d92bf92ab3990665b1a0afe5fa964e49202fb34" Nov 01 22:15:21.742247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.742150 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" event={"ID":"03a3f22e-9730-434a-828b-db0cc6a82ef2","Type":"ContainerDied","Data":"19f9d540713553c07218e87d2cf508412ba3663ee1c5483a92dd04e3b80c6345"} Nov 01 22:15:21.742247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.742209 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f9d540713553c07218e87d2cf508412ba3663ee1c5483a92dd04e3b80c6345" Nov 01 22:15:21.745505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.745310 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:21.760146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.760120 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:21.779676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.769202 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-dfdb11-29892855-dwsxx"] Nov 01 22:15:21.786938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.786874 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerDied","Data":"ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6"} Nov 01 22:15:21.787266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.786688 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerID="ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6" exitCode=60 Nov 01 22:15:21.793255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.790871 5036 scope.go:122] "RemoveContainer" containerID="ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6" Nov 01 22:15:21.793255 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:21.791352 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7_odh-ai-gateway-infra(0b9618b6-869f-457f-ba32-58fde8fcad69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" Nov 01 22:15:21.797768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.796606 5036 generic.go:356] "Generic (PLEG): container finished" podID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerID="032ef36f0ef9a45636f2902112d85eb351b5d70a0923aea011beda82948a59d4" exitCode=60 Nov 01 22:15:21.797768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.796742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerDied","Data":"032ef36f0ef9a45636f2902112d85eb351b5d70a0923aea011beda82948a59d4"} Nov 01 22:15:21.797768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.797228 5036 scope.go:122] "RemoveContainer" containerID="032ef36f0ef9a45636f2902112d85eb351b5d70a0923aea011beda82948a59d4" Nov 01 22:15:21.797768 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:21.797712 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf_odh-ai-gateway-infra(75baa909-f44b-4da4-9756-0dc10f2ad0f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" Nov 01 22:15:21.802569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.800990 5036 generic.go:356] "Generic (PLEG): container finished" podID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerID="a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9" exitCode=60 Nov 01 22:15:21.802569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.801290 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerDied","Data":"a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9"} Nov 01 22:15:21.802569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.801744 5036 scope.go:122] "RemoveContainer" containerID="a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9" Nov 01 22:15:21.802569 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:21.802109 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7_odh-ai-gateway-infra(07f6a84f-1958-4d61-bfde-b61102cdd8d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" Nov 01 22:15:21.806542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.806462 5036 generic.go:356] "Generic (PLEG): container finished" podID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerID="444fe66f371b250908bbab9718c9dc0498c29107f4de1e375a99aa4046d5325c" exitCode=60 Nov 01 22:15:21.806802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.806744 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerDied","Data":"444fe66f371b250908bbab9718c9dc0498c29107f4de1e375a99aa4046d5325c"} Nov 01 22:15:21.807381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.807344 5036 scope.go:122] "RemoveContainer" containerID="444fe66f371b250908bbab9718c9dc0498c29107f4de1e375a99aa4046d5325c" Nov 01 22:15:21.808969 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:21.808914 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx_odh-ai-gateway-infra(7be5bd1f-7a22-44ec-92f8-f156941b97c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" Nov 01 22:15:21.817515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.814340 5036 generic.go:356] "Generic (PLEG): container finished" podID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerID="fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995" exitCode=60 Nov 01 22:15:21.817515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.814412 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerDied","Data":"fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995"} Nov 01 22:15:21.817515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.814817 5036 scope.go:122] "RemoveContainer" containerID="fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995" Nov 01 22:15:21.817515 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:21.815132 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2_odh-ai-gateway-infra(c1448ce4-6179-435e-b9fc-55a5e6b5a7c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" Nov 01 22:15:21.829459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.829422 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:21.920628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.885893 5036 scope.go:122] "RemoveContainer" containerID="cb7f7bdb574fbb11fe38915ca8cfa4ec47daa56b23a4f7319400401cd2d02da3" Nov 01 22:15:21.924952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.924920 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\") pod \"03a3f22e-9730-434a-828b-db0cc6a82ef2\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " Nov 01 22:15:21.925218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.925200 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\" (UniqueName: \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\") pod \"03a3f22e-9730-434a-828b-db0cc6a82ef2\" (UID: \"03a3f22e-9730-434a-828b-db0cc6a82ef2\") " Nov 01 22:15:21.936747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.935181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca" pod "03a3f22e-9730-434a-828b-db0cc6a82ef2" (UID: "03a3f22e-9730-434a-828b-db0cc6a82ef2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:21.944579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.942764 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2" pod "03a3f22e-9730-434a-828b-db0cc6a82ef2" (UID: "03a3f22e-9730-434a-828b-db0cc6a82ef2"). InnerVolumeSpecName "kube-api-access-7dwh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:21.952004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.951981 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a3f22e-9730-434a-828b-db0cc6a82ef2-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:21.952223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:21.952196 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwh2\" (UniqueName: \"kubernetes.io/projected/03a3f22e-9730-434a-828b-db0cc6a82ef2-kube-api-access-7dwh2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.017815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.017784 5036 scope.go:122] "RemoveContainer" containerID="fa40d2df687c628e82a1f47ec9143d32b3fe8733364e02ff88adddd3e5461710" Nov 01 22:15:22.023614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.023583 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:22.030684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.030663 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:22.037829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.037769 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:22.050392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.050350 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:22.053739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.053711 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\" (UniqueName: \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\") pod \"27fb36f2-9ac7-4704-930d-0e25aa26879a\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " Nov 01 22:15:22.053953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.053714 5036 scope.go:122] "RemoveContainer" containerID="fba3baf2fd2cd5e2d4bf1f0f16d43c2db847136ef7cf89aee7835dd1ccac0756" Nov 01 22:15:22.054192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.054043 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\") pod \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " Nov 01 22:15:22.055759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.055736 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\") pod \"27fb36f2-9ac7-4704-930d-0e25aa26879a\" (UID: \"27fb36f2-9ac7-4704-930d-0e25aa26879a\") " Nov 01 22:15:22.056723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.056231 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca" pod "27fb36f2-9ac7-4704-930d-0e25aa26879a" (UID: "27fb36f2-9ac7-4704-930d-0e25aa26879a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:22.113592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.088803 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:22.113592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.095434 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:22.114321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.114290 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\" (UniqueName: \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\") pod \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\" (UID: \"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf\") " Nov 01 22:15:22.117909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.117868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca" pod "a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" (UID: "a7b475c9-75d6-4ce0-9e01-3cf85742b1bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:22.173249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.173198 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn" pod "27fb36f2-9ac7-4704-930d-0e25aa26879a" (UID: "27fb36f2-9ac7-4704-930d-0e25aa26879a"). InnerVolumeSpecName "kube-api-access-jlfrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:22.186940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.186905 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb36f2-9ac7-4704-930d-0e25aa26879a-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.196289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.195698 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:22.206335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.197711 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k" pod "a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" (UID: "a7b475c9-75d6-4ce0-9e01-3cf85742b1bf"). InnerVolumeSpecName "kube-api-access-4xn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:22.206335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.200576 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:22.303579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.303491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\" (UniqueName: \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\") pod \"52dc55de-462e-44a5-a008-750cc7ff28f7\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " Nov 01 22:15:22.303988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.303812 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\") pod \"52dc55de-462e-44a5-a008-750cc7ff28f7\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " Nov 01 22:15:22.304317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.304298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\") pod \"52dc55de-462e-44a5-a008-750cc7ff28f7\" (UID: \"52dc55de-462e-44a5-a008-750cc7ff28f7\") " Nov 01 22:15:22.305536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.305501 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfrn\" (UniqueName: \"kubernetes.io/projected/27fb36f2-9ac7-4704-930d-0e25aa26879a-kube-api-access-jlfrn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.305703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.305685 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.305828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.305812 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xn9k\" (UniqueName: \"kubernetes.io/projected/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf-kube-api-access-4xn9k\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.306244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.306226 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca" pod "52dc55de-462e-44a5-a008-750cc7ff28f7" (UID: "52dc55de-462e-44a5-a008-750cc7ff28f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:22.309180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.306796 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs" pod "52dc55de-462e-44a5-a008-750cc7ff28f7" (UID: "52dc55de-462e-44a5-a008-750cc7ff28f7"). InnerVolumeSpecName "kube-api-access-n7mcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:22.314301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.314273 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca" pod "52dc55de-462e-44a5-a008-750cc7ff28f7" (UID: "52dc55de-462e-44a5-a008-750cc7ff28f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:22.336231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.336202 5036 scope.go:122] "RemoveContainer" containerID="ea0f0eeca21843389bb8ed726311a3876a36576a6beb6a01d73074f0bea73afb" Nov 01 22:15:22.407996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.407879 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mcs\" (UniqueName: \"kubernetes.io/projected/52dc55de-462e-44a5-a008-750cc7ff28f7-kube-api-access-n7mcs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.407996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.407919 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.407996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.407940 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dc55de-462e-44a5-a008-750cc7ff28f7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.499236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.499141 5036 scope.go:122] "RemoveContainer" containerID="5939bd8f6cae4742ff603c65e6a47b72ca12bf8d38e87bcb284da70f92f393bb" Nov 01 22:15:22.509014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.508982 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" path="/var/lib/kubelet/pods/14f13ed8-24f7-493b-8c7f-a8d632156d69/volumes" Nov 01 22:15:22.511322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.511292 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" path="/var/lib/kubelet/pods/a1d159d2-492c-4e80-b0d4-2bb118563a87/volumes" Nov 01 22:15:22.663123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.663057 5036 scope.go:122] "RemoveContainer" containerID="780b7ab832706086e91455067567ddda1282d59d78e71d2e925f8782cb215a58" Nov 01 22:15:22.667208 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:22.667179 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749a065c_6d82_4094_a036_d47d93529fd2.slice/crio-7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257 WatchSource:0}: Error finding container 7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257: Status 404 returned error can't find the container with id 7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257 Nov 01 22:15:22.673296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.673268 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z"] Nov 01 22:15:22.738951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.738867 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:22.852665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.837929 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\") pod \"24488105-d393-4f74-bf75-17d186b794a2\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " Nov 01 22:15:22.852665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.838109 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\" (UniqueName: \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\") pod \"24488105-d393-4f74-bf75-17d186b794a2\" (UID: \"24488105-d393-4f74-bf75-17d186b794a2\") " Nov 01 22:15:22.852665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.845657 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca" pod "24488105-d393-4f74-bf75-17d186b794a2" (UID: "24488105-d393-4f74-bf75-17d186b794a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:22.887117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.884817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:22.891321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.891287 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:22.893555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.892086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc" pod "24488105-d393-4f74-bf75-17d186b794a2" (UID: "24488105-d393-4f74-bf75-17d186b794a2"). InnerVolumeSpecName "kube-api-access-9rqhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:22.895715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.895685 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:22.900310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.845933 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24488105-d393-4f74-bf75-17d186b794a2-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.901315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.901294 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:22.956163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.956134 5036 scope.go:122] "RemoveContainer" containerID="00d7e843edb924f06d4fb60890885cd008f1363573189aca00ac1e591d792eba" Nov 01 22:15:22.999674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.984043 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:23.000011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:22.999978 5036 generic.go:356] "Generic (PLEG): container finished" podID="380a1985-8962-4447-9d6a-4a399646e5eb" containerID="a01e171acc4a63fa46817cd1317a4c839727908941dd9eecb685f760eec77dd3" exitCode=60 Nov 01 22:15:23.000371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.000206 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" event={"ID":"380a1985-8962-4447-9d6a-4a399646e5eb","Type":"ContainerDied","Data":"a01e171acc4a63fa46817cd1317a4c839727908941dd9eecb685f760eec77dd3"} Nov 01 22:15:23.001804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.001273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\" (UniqueName: \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\") pod \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " Nov 01 22:15:23.001804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.001631 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\") pod \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\" (UID: \"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef\") " Nov 01 22:15:23.001804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.001689 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\") pod \"86fc72a8-060e-4573-b43c-50c40521abf0\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " Nov 01 22:15:23.001804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.001737 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\") pod \"699f6c9b-412f-4987-85e9-3308e01ae622\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " Nov 01 22:15:23.003433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.002130 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\" (UniqueName: \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\") pod \"699f6c9b-412f-4987-85e9-3308e01ae622\" (UID: \"699f6c9b-412f-4987-85e9-3308e01ae622\") " Nov 01 22:15:23.003433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.002206 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\" (UniqueName: \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\") pod \"86fc72a8-060e-4573-b43c-50c40521abf0\" (UID: \"86fc72a8-060e-4573-b43c-50c40521abf0\") " Nov 01 22:15:23.003433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.003108 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqhc\" (UniqueName: \"kubernetes.io/projected/24488105-d393-4f74-bf75-17d186b794a2-kube-api-access-9rqhc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.003966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.003952 5036 scope.go:122] "RemoveContainer" containerID="a01e171acc4a63fa46817cd1317a4c839727908941dd9eecb685f760eec77dd3" Nov 01 22:15:23.004587 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:23.004565 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689_odh-ai-gateway-infra(380a1985-8962-4447-9d6a-4a399646e5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" Nov 01 22:15:23.005566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.005529 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca" pod "86fc72a8-060e-4573-b43c-50c40521abf0" (UID: "86fc72a8-060e-4573-b43c-50c40521abf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:23.006790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.006768 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca" pod "699f6c9b-412f-4987-85e9-3308e01ae622" (UID: "699f6c9b-412f-4987-85e9-3308e01ae622"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:23.007856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.007746 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca" pod "c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" (UID: "c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:23.014703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.014675 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" event={"ID":"86fc72a8-060e-4573-b43c-50c40521abf0","Type":"ContainerDied","Data":"07a2ab4ae00aa983c42cac6603ec8825c48f54e7296cbc0c03a7819b82363c80"} Nov 01 22:15:23.014938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.014921 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9" Nov 01 22:15:23.039790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.039545 5036 scope.go:122] "RemoveContainer" containerID="fc46bc307339b44cfc281db370e99283a4c2ebb0809364475572e69d5753cfb8" Nov 01 22:15:23.045816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.045792 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" Nov 01 22:15:23.049590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.046700 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv" event={"ID":"27fb36f2-9ac7-4704-930d-0e25aa26879a","Type":"ContainerDied","Data":"f4c5ad9ec631188dbab957ca7644173ccbc8d9453dfc74c9c82f18958f88020f"} Nov 01 22:15:23.054465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.054237 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" Nov 01 22:15:23.054465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.054386 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t" event={"ID":"699f6c9b-412f-4987-85e9-3308e01ae622","Type":"ContainerDied","Data":"4f6e9898ee5b931ca6e5826822912c41ec36eb51a6afad173c5987c694882dc6"} Nov 01 22:15:23.058351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.057491 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" event={"ID":"52dc55de-462e-44a5-a008-750cc7ff28f7","Type":"ContainerDied","Data":"20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4"} Nov 01 22:15:23.058351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.057541 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20162ae7a2123b342f0014b3e2a6b8721adf6845e61fa9448e80e692a54ed5b4" Nov 01 22:15:23.058351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.057617 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6" Nov 01 22:15:23.062379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.061290 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" event={"ID":"c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef","Type":"ContainerDied","Data":"e55810f0a445f4529abcfa42de7301080b681b27eac48be4669902e0637a248c"} Nov 01 22:15:23.062379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.061357 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc" Nov 01 22:15:23.065612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.063940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" event={"ID":"24488105-d393-4f74-bf75-17d186b794a2","Type":"ContainerDied","Data":"4bcadf24fb96c96df16abf3db400df09e2cfc33676d14cc14d6174d1152d64ea"} Nov 01 22:15:23.065612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.064044 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm" Nov 01 22:15:23.068020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.067958 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" event={"ID":"749a065c-6d82-4094-a036-d47d93529fd2","Type":"ContainerStarted","Data":"7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257"} Nov 01 22:15:23.071470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.070930 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg" Nov 01 22:15:23.071470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.071094 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" event={"ID":"a7b475c9-75d6-4ce0-9e01-3cf85742b1bf","Type":"ContainerDied","Data":"4fff79e15fa0a0cd0f6a9e2a554d294712982aa995ef08466e0d49bbe32f3662"} Nov 01 22:15:23.071470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.071237 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg" Nov 01 22:15:23.087644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.087615 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k" pod "c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" (UID: "c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef"). InnerVolumeSpecName "kube-api-access-zv64k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:23.090069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.090042 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk" pod "699f6c9b-412f-4987-85e9-3308e01ae622" (UID: "699f6c9b-412f-4987-85e9-3308e01ae622"). InnerVolumeSpecName "kube-api-access-24wbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:23.090429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.090363 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms" pod "86fc72a8-060e-4573-b43c-50c40521abf0" (UID: "86fc72a8-060e-4573-b43c-50c40521abf0"). InnerVolumeSpecName "kube-api-access-2tjms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:23.112915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.112884 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv64k\" (UniqueName: \"kubernetes.io/projected/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-kube-api-access-zv64k\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113063 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113176 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc72a8-060e-4573-b43c-50c40521abf0-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113268 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f6c9b-412f-4987-85e9-3308e01ae622-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113360 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24wbk\" (UniqueName: \"kubernetes.io/projected/699f6c9b-412f-4987-85e9-3308e01ae622-kube-api-access-24wbk\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113487 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjms\" (UniqueName: \"kubernetes.io/projected/86fc72a8-060e-4573-b43c-50c40521abf0-kube-api-access-2tjms\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.113599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.113134 5036 scope.go:122] "RemoveContainer" containerID="4069cfdd9ff3770941e38ab06e3deaa9ad5c4e365f3d5410a6acb3ccb0b5e248" Nov 01 22:15:23.163582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.163481 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:23.182360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.172051 5036 scope.go:122] "RemoveContainer" containerID="166c764ff236360aad1c580dc1a7f0218f7b743a41addd454cbf1f848cfb2e0d" Nov 01 22:15:23.182360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.172720 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-de0cba-29892855-ppzqm"] Nov 01 22:15:23.197640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.197604 5036 scope.go:122] "RemoveContainer" containerID="40431933a180986ea4338421cd024f4fc93317134fed06064ae9888055d5d01a" Nov 01 22:15:23.238380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.237748 5036 scope.go:122] "RemoveContainer" containerID="b35d88f053d012138621c544f41848d271223f7dbb7bf351a8d150f5e932e51e" Nov 01 22:15:23.285140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.285117 5036 scope.go:122] "RemoveContainer" containerID="07c30f5af926cab08303688873ccb7c6c1b22a8f78d69fd98478fa444ff2fdcd" Nov 01 22:15:23.390322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.390293 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:23.433040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.432680 5036 scope.go:122] "RemoveContainer" containerID="5de192c7f764e7b008604521e80101d976331c5f5897adb2113bc4ad06849581" Nov 01 22:15:23.433040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.432807 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-830149-29892855-j46tg"] Nov 01 22:15:23.442586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.442551 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:23.459928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.459891 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892855-swbgv"] Nov 01 22:15:23.475601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.475572 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:23.535365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.535339 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:23.541781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.541758 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892855-xs7pg"] Nov 01 22:15:23.621088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.620984 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:23.662555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.662521 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8483c2-29892855-lvbh9"] Nov 01 22:15:23.688727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.688696 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:23.702035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.702006 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d6e25f-29892855-n5zgc"] Nov 01 22:15:23.707099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.707062 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\") pod \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " Nov 01 22:15:23.708132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.708101 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\" (UniqueName: \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\") pod \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\" (UID: \"7be5bd1f-7a22-44ec-92f8-f156941b97c2\") " Nov 01 22:15:23.719476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.707656 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca" pod "7be5bd1f-7a22-44ec-92f8-f156941b97c2" (UID: "7be5bd1f-7a22-44ec-92f8-f156941b97c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:23.732841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.732797 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf" pod "7be5bd1f-7a22-44ec-92f8-f156941b97c2" (UID: "7be5bd1f-7a22-44ec-92f8-f156941b97c2"). InnerVolumeSpecName "kube-api-access-xbksf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:23.917740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.915250 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:23.918720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.918669 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbksf\" (UniqueName: \"kubernetes.io/projected/7be5bd1f-7a22-44ec-92f8-f156941b97c2-kube-api-access-xbksf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.918895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.918878 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be5bd1f-7a22-44ec-92f8-f156941b97c2-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:23.958665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.958635 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:23.966935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.966908 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f2760d-29892855-9489t"] Nov 01 22:15:23.975988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.975634 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:23.982277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.981637 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:23.988799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.988709 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:23.999524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:23.999362 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:24.021954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.021922 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\" (UniqueName: \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\") pod \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " Nov 01 22:15:24.022132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022012 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\") pod \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " Nov 01 22:15:24.022132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022114 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\") pod \"0b9618b6-869f-457f-ba32-58fde8fcad69\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " Nov 01 22:15:24.022253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022145 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\") pod \"0b9618b6-869f-457f-ba32-58fde8fcad69\" (UID: \"0b9618b6-869f-457f-ba32-58fde8fcad69\") " Nov 01 22:15:24.022253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022205 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\") pod \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\" (UID: \"07f6a84f-1958-4d61-bfde-b61102cdd8d7\") " Nov 01 22:15:24.022366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022281 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\") pod \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " Nov 01 22:15:24.022366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\" (UniqueName: \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\") pod \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\" (UID: \"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9\") " Nov 01 22:15:24.022517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.022499 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\" (UniqueName: \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\") pod \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\" (UID: \"75baa909-f44b-4da4-9756-0dc10f2ad0f6\") " Nov 01 22:15:24.024357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.024253 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca" pod "0b9618b6-869f-457f-ba32-58fde8fcad69" (UID: "0b9618b6-869f-457f-ba32-58fde8fcad69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:24.028361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.027461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca" pod "75baa909-f44b-4da4-9756-0dc10f2ad0f6" (UID: "75baa909-f44b-4da4-9756-0dc10f2ad0f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:24.028361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.027838 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca" pod "07f6a84f-1958-4d61-bfde-b61102cdd8d7" (UID: "07f6a84f-1958-4d61-bfde-b61102cdd8d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:24.034879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.031155 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk" pod "c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" (UID: "c1448ce4-6179-435e-b9fc-55a5e6b5a7c9"). InnerVolumeSpecName "kube-api-access-w58qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:24.035471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.035137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca" pod "c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" (UID: "c1448ce4-6179-435e-b9fc-55a5e6b5a7c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:24.039542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.039392 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc" pod "0b9618b6-869f-457f-ba32-58fde8fcad69" (UID: "0b9618b6-869f-457f-ba32-58fde8fcad69"). InnerVolumeSpecName "kube-api-access-s8smc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:24.040108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.040058 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f" pod "75baa909-f44b-4da4-9756-0dc10f2ad0f6" (UID: "75baa909-f44b-4da4-9756-0dc10f2ad0f6"). InnerVolumeSpecName "kube-api-access-jzh7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:24.048585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.048557 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr" pod "07f6a84f-1958-4d61-bfde-b61102cdd8d7" (UID: "07f6a84f-1958-4d61-bfde-b61102cdd8d7"). InnerVolumeSpecName "kube-api-access-qtwzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:24.075409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.075372 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:24.093976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.093939 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" event={"ID":"749a065c-6d82-4094-a036-d47d93529fd2","Type":"ContainerStarted","Data":"c1cb896eb4f3857e298b2f768cc8ee413b0c658f2422da834fa3bed7731f1639"} Nov 01 22:15:24.114883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.114847 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" event={"ID":"7be5bd1f-7a22-44ec-92f8-f156941b97c2","Type":"ContainerDied","Data":"858040f4a2151ef774ef2816f930ac32c6ff0609d642a460dbcbd028c47b82f9"} Nov 01 22:15:24.115040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.114915 5036 scope.go:122] "RemoveContainer" containerID="444fe66f371b250908bbab9718c9dc0498c29107f4de1e375a99aa4046d5325c" Nov 01 22:15:24.115675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.115124 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx" Nov 01 22:15:24.130844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.127860 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzh7f\" (UniqueName: \"kubernetes.io/projected/75baa909-f44b-4da4-9756-0dc10f2ad0f6-kube-api-access-jzh7f\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.131119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.131079 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwzr\" (UniqueName: \"kubernetes.io/projected/07f6a84f-1958-4d61-bfde-b61102cdd8d7-kube-api-access-qtwzr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.131277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.131259 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.132283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.132005 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/0b9618b6-869f-457f-ba32-58fde8fcad69-kube-api-access-s8smc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.132478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.132459 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9618b6-869f-457f-ba32-58fde8fcad69-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.132921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.132903 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f6a84f-1958-4d61-bfde-b61102cdd8d7-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.133072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.133056 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75baa909-f44b-4da4-9756-0dc10f2ad0f6-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.133195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.133178 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58qk\" (UniqueName: \"kubernetes.io/projected/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9-kube-api-access-w58qk\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.135705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.135682 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" Nov 01 22:15:24.135810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.135706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2" event={"ID":"c1448ce4-6179-435e-b9fc-55a5e6b5a7c9","Type":"ContainerDied","Data":"8c1164b96e9f2f66fe26a41ca38f3af4fa5bee39e0be151d817a2203106c4552"} Nov 01 22:15:24.143496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.140659 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" podStartSLOduration=4.140642833 podStartE2EDuration="4.140642833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:24.134696374 +0000 UTC m=+1877.827676861" watchObservedRunningTime="2026-11-01 22:15:24.140642833 +0000 UTC m=+1877.833623319" Nov 01 22:15:24.148677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.148562 5036 scope.go:122] "RemoveContainer" containerID="fbe0c9cc6297c6d7e5f2b0132669af38ede5742c8433b5f3afe6ec9537cd1995" Nov 01 22:15:24.149784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.149700 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" Nov 01 22:15:24.150256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.150193 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7" event={"ID":"0b9618b6-869f-457f-ba32-58fde8fcad69","Type":"ContainerDied","Data":"8c5c68f5a4ce7758f8aefd78230ef2dd23dd539db1071807caa04a3c3084296b"} Nov 01 22:15:24.152735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.152717 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" Nov 01 22:15:24.154891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.152803 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf" event={"ID":"75baa909-f44b-4da4-9756-0dc10f2ad0f6","Type":"ContainerDied","Data":"772651eca52100ca0b6e5619575b55d040df7e645b99f3cc3148b4f3445c9a40"} Nov 01 22:15:24.155972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.155267 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" Nov 01 22:15:24.155972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.155533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7" event={"ID":"07f6a84f-1958-4d61-bfde-b61102cdd8d7","Type":"ContainerDied","Data":"db881e8bec0b6237f7e623bb5d06afad504af0864fb7db9c219946098272b856"} Nov 01 22:15:24.165475 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:24.165416 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9618b6_869f_457f_ba32_58fde8fcad69.slice/crio-8c5c68f5a4ce7758f8aefd78230ef2dd23dd539db1071807caa04a3c3084296b\": RecentStats: unable to find data in memory cache]" Nov 01 22:15:24.192712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.192682 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:24.193844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.193820 5036 scope.go:122] "RemoveContainer" containerID="ffe99ccb520e842b0c956b3e245958056497cbfbb894fbb34fa98cf4a7aae3c6" Nov 01 22:15:24.204700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.204672 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3e300c-29892855-8blx2"] Nov 01 22:15:24.254520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.254482 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:24.263563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.263474 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-487639-29892855-8rkvx"] Nov 01 22:15:24.269982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.269961 5036 scope.go:122] "RemoveContainer" containerID="032ef36f0ef9a45636f2902112d85eb351b5d70a0923aea011beda82948a59d4" Nov 01 22:15:24.280192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.280168 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:24.290966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.290945 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-070e3d-29892855-wbjz7"] Nov 01 22:15:24.298082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.298047 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:24.304275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.304251 5036 scope.go:122] "RemoveContainer" containerID="a0c884383784a82dc55f3fe16bbb3bd3ef492ea6fa27369acc789bb79abd9dc9" Nov 01 22:15:24.326569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.326504 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-17957f-29892855-z6ztf"] Nov 01 22:15:24.334359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.334337 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:24.370015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.369977 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892855-wtjw7"] Nov 01 22:15:24.455460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.453753 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" path="/var/lib/kubelet/pods/03a3f22e-9730-434a-828b-db0cc6a82ef2/volumes" Nov 01 22:15:24.455460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.454420 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" path="/var/lib/kubelet/pods/07f6a84f-1958-4d61-bfde-b61102cdd8d7/volumes" Nov 01 22:15:24.455460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.455097 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" path="/var/lib/kubelet/pods/0b9618b6-869f-457f-ba32-58fde8fcad69/volumes" Nov 01 22:15:24.457238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.457212 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24488105-d393-4f74-bf75-17d186b794a2" path="/var/lib/kubelet/pods/24488105-d393-4f74-bf75-17d186b794a2/volumes" Nov 01 22:15:24.457907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.457886 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" path="/var/lib/kubelet/pods/27fb36f2-9ac7-4704-930d-0e25aa26879a/volumes" Nov 01 22:15:24.458554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.458534 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" path="/var/lib/kubelet/pods/699f6c9b-412f-4987-85e9-3308e01ae622/volumes" Nov 01 22:15:24.458875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.458854 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:24.459656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.459636 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" path="/var/lib/kubelet/pods/75baa909-f44b-4da4-9756-0dc10f2ad0f6/volumes" Nov 01 22:15:24.460287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.460267 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" path="/var/lib/kubelet/pods/7be5bd1f-7a22-44ec-92f8-f156941b97c2/volumes" Nov 01 22:15:24.460999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.460978 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" path="/var/lib/kubelet/pods/86fc72a8-060e-4573-b43c-50c40521abf0/volumes" Nov 01 22:15:24.462697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.462674 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" path="/var/lib/kubelet/pods/a7b475c9-75d6-4ce0-9e01-3cf85742b1bf/volumes" Nov 01 22:15:24.463541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.463503 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" path="/var/lib/kubelet/pods/c1448ce4-6179-435e-b9fc-55a5e6b5a7c9/volumes" Nov 01 22:15:24.464269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.464241 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" path="/var/lib/kubelet/pods/c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef/volumes" Nov 01 22:15:24.497474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.481535 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:24.554496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.554464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\") pod \"380a1985-8962-4447-9d6a-4a399646e5eb\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " Nov 01 22:15:24.554624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.554592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\" (UniqueName: \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\") pod \"380a1985-8962-4447-9d6a-4a399646e5eb\" (UID: \"380a1985-8962-4447-9d6a-4a399646e5eb\") " Nov 01 22:15:24.556132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.556088 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca" pod "380a1985-8962-4447-9d6a-4a399646e5eb" (UID: "380a1985-8962-4447-9d6a-4a399646e5eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:24.558204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.558171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx" pod "380a1985-8962-4447-9d6a-4a399646e5eb" (UID: "380a1985-8962-4447-9d6a-4a399646e5eb"). InnerVolumeSpecName "kube-api-access-vcfjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:24.622511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.622285 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:24.656899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.656868 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a1985-8962-4447-9d6a-4a399646e5eb-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.656899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.656898 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfjx\" (UniqueName: \"kubernetes.io/projected/380a1985-8962-4447-9d6a-4a399646e5eb-kube-api-access-vcfjx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:24.841514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:24.841481 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:25.208633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.208553 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" Nov 01 22:15:25.209087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.208561 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689" event={"ID":"380a1985-8962-4447-9d6a-4a399646e5eb","Type":"ContainerDied","Data":"4385fa3d8ed4489d0a4e40f82a2509d8bd993e54b0853c3c05465d5dc30ae829"} Nov 01 22:15:25.209087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.208723 5036 scope.go:122] "RemoveContainer" containerID="a01e171acc4a63fa46817cd1317a4c839727908941dd9eecb685f760eec77dd3" Nov 01 22:15:25.215370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.215344 5036 generic.go:356] "Generic (PLEG): container finished" podID="749a065c-6d82-4094-a036-d47d93529fd2" containerID="c1cb896eb4f3857e298b2f768cc8ee413b0c658f2422da834fa3bed7731f1639" exitCode=0 Nov 01 22:15:25.215566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.215385 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" event={"ID":"749a065c-6d82-4094-a036-d47d93529fd2","Type":"ContainerDied","Data":"c1cb896eb4f3857e298b2f768cc8ee413b0c658f2422da834fa3bed7731f1639"} Nov 01 22:15:25.251749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.251722 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:25.260365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.260338 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d0a5de-29892855-n7689"] Nov 01 22:15:25.671096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.671059 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:25.882486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:25.878179 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:26.460283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.460240 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" path="/var/lib/kubelet/pods/380a1985-8962-4447-9d6a-4a399646e5eb/volumes" Nov 01 22:15:26.474484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.474458 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:26.541074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.541033 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\" (UniqueName: \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\") pod \"749a065c-6d82-4094-a036-d47d93529fd2\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " Nov 01 22:15:26.541272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.541110 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\") pod \"749a065c-6d82-4094-a036-d47d93529fd2\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " Nov 01 22:15:26.541272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.541256 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\") pod \"749a065c-6d82-4094-a036-d47d93529fd2\" (UID: \"749a065c-6d82-4094-a036-d47d93529fd2\") " Nov 01 22:15:26.541795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.541553 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca" pod "749a065c-6d82-4094-a036-d47d93529fd2" (UID: "749a065c-6d82-4094-a036-d47d93529fd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:26.542501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.542475 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:26.543849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.543814 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq" pod "749a065c-6d82-4094-a036-d47d93529fd2" (UID: "749a065c-6d82-4094-a036-d47d93529fd2"). InnerVolumeSpecName "kube-api-access-89pnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:26.544003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.543976 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca" pod "749a065c-6d82-4094-a036-d47d93529fd2" (UID: "749a065c-6d82-4094-a036-d47d93529fd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:26.645032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.644998 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pnq\" (UniqueName: \"kubernetes.io/projected/749a065c-6d82-4094-a036-d47d93529fd2-kube-api-access-89pnq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:26.645032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.645033 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749a065c-6d82-4094-a036-d47d93529fd2-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:26.757941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:26.757905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:27.256918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:27.256888 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" Nov 01 22:15:27.257102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:27.256910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z" event={"ID":"749a065c-6d82-4094-a036-d47d93529fd2","Type":"ContainerDied","Data":"7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257"} Nov 01 22:15:27.257102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:27.256969 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7086aa17d023481d655c187a6cc91b4193b8af954e99e434fecadd98a22b0257" Nov 01 22:15:27.658406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:27.658366 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:28.418422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:28.418384 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:29.282036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:29.281966 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:30.038010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:30.037976 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:30.634784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:30.634738 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" containerID="cri-o://9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf" gracePeriod=45 Nov 01 22:15:30.645354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:30.645321 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.208:8000/ready\": dial tcp 10.132.0.208:8000: connect: connection refused" Nov 01 22:15:30.837467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:30.837004 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:31.037219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.037195 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:15:31.173330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173294 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173369 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\" (UniqueName: \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173411 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173489 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173585 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:31.173945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173748 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\") pod \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\" (UID: \"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad\") " Nov 01 22:15:31.173945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:31.174037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.173970 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:31.175078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.175050 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.175199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.175082 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.175199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.175097 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.176251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.176227 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:31.176394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.176366 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:31.176511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.176415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn" pod "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" (UID: "151bc2b8-bc39-478c-8a20-ea4e1c20e7ad"). InnerVolumeSpecName "kube-api-access-xkjbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:31.276831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.276798 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjbn\" (UniqueName: \"kubernetes.io/projected/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-kube-api-access-xkjbn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.276956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.276838 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.276956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.276855 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:31.315282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.315249 5036 generic.go:356] "Generic (PLEG): container finished" podID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerID="9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf" exitCode=0 Nov 01 22:15:31.315429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.315312 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" event={"ID":"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad","Type":"ContainerDied","Data":"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf"} Nov 01 22:15:31.315429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.315347 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" event={"ID":"151bc2b8-bc39-478c-8a20-ea4e1c20e7ad","Type":"ContainerDied","Data":"2c190de2ecdbc5fe78e6e7a1a147adb770fd29172b34c4ac8d8ff6d8cdc57935"} Nov 01 22:15:31.315429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.315365 5036 scope.go:122] "RemoveContainer" containerID="9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf" Nov 01 22:15:31.315660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.315548 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m" Nov 01 22:15:31.335431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.335405 5036 scope.go:122] "RemoveContainer" containerID="9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf" Nov 01 22:15:31.335744 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:31.335717 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf\": container with ID starting with 9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf not found: ID does not exist" containerID="9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf" Nov 01 22:15:31.335807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.335760 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf"} err="failed to get container status \"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf\": rpc error: code = NotFound desc = could not find container \"9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf\": container with ID starting with 9493c2c34c1ef834ac116d82afd97413fc749f8c23cf37dfe5c06993a592c5bf not found: ID does not exist" Nov 01 22:15:31.349314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.349287 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:15:31.353787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.353764 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d0a5de/test-model-d0a5de-kserve-69fb759c49-n874m"] Nov 01 22:15:31.676463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:31.676353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:32.475356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.475316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:32.489183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.482245 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" path="/var/lib/kubelet/pods/151bc2b8-bc39-478c-8a20-ea4e1c20e7ad/volumes" Nov 01 22:15:32.489183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.482783 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:15:32.640470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.639743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:32.961277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.961235 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh"] Nov 01 22:15:32.962043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962022 5036 cpu_manager.go:414] "containerMap: removing container" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.962043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962041 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962053 5036 cpu_manager.go:414] "containerMap: removing container" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962061 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962072 5036 cpu_manager.go:414] "containerMap: removing container" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962077 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962083 5036 cpu_manager.go:414] "containerMap: removing container" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962088 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962094 5036 cpu_manager.go:414] "containerMap: removing container" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962099 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962108 5036 cpu_manager.go:414] "containerMap: removing container" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962113 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962120 5036 cpu_manager.go:414] "containerMap: removing container" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962125 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962131 5036 cpu_manager.go:414] "containerMap: removing container" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962136 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962143 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962147 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962155 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962160 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962168 5036 cpu_manager.go:414] "containerMap: removing container" podUID="749a065c-6d82-4094-a036-d47d93529fd2" containerName="revoke-keys" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962172 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749a065c-6d82-4094-a036-d47d93529fd2" containerName="revoke-keys" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962184 5036 cpu_manager.go:414] "containerMap: removing container" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962188 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962197 5036 cpu_manager.go:414] "containerMap: removing container" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962202 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962209 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962214 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962222 5036 cpu_manager.go:414] "containerMap: removing container" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962226 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962233 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962238 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962247 5036 cpu_manager.go:414] "containerMap: removing container" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962252 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f13ed8-24f7-493b-8c7f-a8d632156d69" containerName="cleanup" Nov 01 22:15:32.962238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962261 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962266 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962273 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962277 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962287 5036 cpu_manager.go:414] "containerMap: removing container" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962291 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962300 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962304 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962312 5036 cpu_manager.go:414] "containerMap: removing container" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962316 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962325 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962329 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962337 5036 cpu_manager.go:414] "containerMap: removing container" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962341 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962348 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962353 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962359 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962363 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962372 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962376 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962383 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962387 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962395 5036 cpu_manager.go:414] "containerMap: removing container" podUID="52dc55de-462e-44a5-a008-750cc7ff28f7" containerName="revoke-keys" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962399 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52dc55de-462e-44a5-a008-750cc7ff28f7" containerName="revoke-keys" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962406 5036 cpu_manager.go:414] "containerMap: removing container" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962410 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962427 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962431 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962454 5036 cpu_manager.go:414] "containerMap: removing container" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962460 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962471 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962475 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962484 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962489 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962615 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962624 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962631 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962637 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962643 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962650 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962656 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962661 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962667 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="151bc2b8-bc39-478c-8a20-ea4e1c20e7ad" containerName="main" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962674 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962679 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962686 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d159d2-492c-4e80-b0d4-2bb118563a87" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962691 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962697 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962703 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962708 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="52dc55de-462e-44a5-a008-750cc7ff28f7" containerName="revoke-keys" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962714 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962722 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962727 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962733 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962738 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b475c9-75d6-4ce0-9e01-3cf85742b1bf" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962745 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962750 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962755 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962761 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962768 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962775 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962780 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962785 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962791 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962802 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962808 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962814 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="749a065c-6d82-4094-a036-d47d93529fd2" containerName="revoke-keys" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962821 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962827 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:32.964036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.962833 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be5bd1f-7a22-44ec-92f8-f156941b97c2" containerName="cleanup" Nov 01 22:15:32.968201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.968180 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:32.974462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.974047 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:32.999351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:32.999321 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:15:33.023794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.023063 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh"] Nov 01 22:15:33.041273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.041236 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.041654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.041629 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.041789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.041767 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4d55\" (UniqueName: \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.143606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.143561 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.143798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.143673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4d55\" (UniqueName: \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.143798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.143753 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.144685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.144657 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.146259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.146236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.156625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.156597 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4d55\" (UniqueName: \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\") pod \"maas-api-revoke-sub-f93c26cb-rlnmh\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.324123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.324090 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:33.332142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.331832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:33.626505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.625509 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:15:33.668467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.667522 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh"] Nov 01 22:15:33.680538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.670181 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:33.680538 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:33.678315 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e9064d_fd73_4909_925c_70aa883720ac.slice/crio-3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4 WatchSource:0}: Error finding container 3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4: Status 404 returned error can't find the container with id 3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4 Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.966509 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7"] Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971148 5036 cpu_manager.go:414] "containerMap: removing container" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971190 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380a1985-8962-4447-9d6a-4a399646e5eb" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971221 5036 cpu_manager.go:414] "containerMap: removing container" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971229 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971245 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971266 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971302 5036 cpu_manager.go:414] "containerMap: removing container" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:33.979533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.971311 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981805 5036 cpu_manager.go:414] "containerMap: removing container" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981851 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981951 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981959 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1448ce4-6179-435e-b9fc-55a5e6b5a7c9" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981973 5036 cpu_manager.go:414] "containerMap: removing container" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.981980 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb36f2-9ac7-4704-930d-0e25aa26879a" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982005 5036 cpu_manager.go:414] "containerMap: removing container" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982011 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f6a84f-1958-4d61-bfde-b61102cdd8d7" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982029 5036 cpu_manager.go:414] "containerMap: removing container" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982036 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75baa909-f44b-4da4-9756-0dc10f2ad0f6" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982052 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982072 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982373 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9618b6-869f-457f-ba32-58fde8fcad69" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982404 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="24488105-d393-4f74-bf75-17d186b794a2" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982417 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a3f22e-9730-434a-828b-db0cc6a82ef2" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982433 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="699f6c9b-412f-4987-85e9-3308e01ae622" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982506 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc72a8-060e-4573-b43c-50c40521abf0" containerName="cleanup" Nov 01 22:15:33.989556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:33.982558 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f64d11-50ba-4705-86f7-bdaa1c6ae5ef" containerName="cleanup" Nov 01 22:15:34.103873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.103761 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.141829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.133310 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:34.255057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.201356 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf7t\" (UniqueName: \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.255057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.201602 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.255057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.201716 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.326500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.306438 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf7t\" (UniqueName: \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.326500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.306662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.326500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.306772 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.351827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.351779 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.368520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.360502 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7"] Nov 01 22:15:34.427578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.427499 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:15:34.461252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.461214 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.472068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.472003 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf7t\" (UniqueName: \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\") pod \"maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.506717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.506629 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" event={"ID":"60e9064d-fd73-4909-925c-70aa883720ac","Type":"ContainerStarted","Data":"3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4"} Nov 01 22:15:34.507060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.507040 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb"] Nov 01 22:15:34.513773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.513752 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.518582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.518559 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb"] Nov 01 22:15:34.545758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.545728 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzzm\" (UniqueName: \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.545962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.545940 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.546186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.546163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.565981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.565951 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:34.648471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.641314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:34.662683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.650880 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.662683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.650988 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzzm\" (UniqueName: \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.662683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.651149 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.662683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.652000 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.678088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.678055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzzm\" (UniqueName: \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.694472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.679288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.839452 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.839419 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:34.929735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.929313 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" containerID="cri-o://c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b" gracePeriod=45 Nov 01 22:15:34.988904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:34.987123 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7"] Nov 01 22:15:35.149934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.149879 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb"] Nov 01 22:15:35.182178 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:35.182130 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c18c69_1d1b_4912_a455_50423b6c61a3.slice/crio-67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d WatchSource:0}: Error finding container 67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d: Status 404 returned error can't find the container with id 67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d Nov 01 22:15:35.382356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.382330 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:15:35.431805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.431762 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:35.469857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.469821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.470162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.469922 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\" (UniqueName: \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.470292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.470155 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.470375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.470315 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.470375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.470369 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.470515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.470425 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\") pod \"c194ac77-f017-48ed-916f-bd7425f9dcb9\" (UID: \"c194ac77-f017-48ed-916f-bd7425f9dcb9\") " Nov 01 22:15:35.472203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.472172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:35.486793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.473091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:35.491699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.490135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:35.518563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.516748 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:35.518563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.516834 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:35.518563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.516943 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt" pod "c194ac77-f017-48ed-916f-bd7425f9dcb9" (UID: "c194ac77-f017-48ed-916f-bd7425f9dcb9"). InnerVolumeSpecName "kube-api-access-g4gxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:35.542391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.542308 5036 generic.go:356] "Generic (PLEG): container finished" podID="60e9064d-fd73-4909-925c-70aa883720ac" containerID="5da209411034c8219879055d1d20d48c3096666f117e3f77f2fcdece009bd04b" exitCode=0 Nov 01 22:15:35.542644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.542618 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" event={"ID":"60e9064d-fd73-4909-925c-70aa883720ac","Type":"ContainerDied","Data":"5da209411034c8219879055d1d20d48c3096666f117e3f77f2fcdece009bd04b"} Nov 01 22:15:35.545016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.544852 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" Nov 01 22:15:35.545016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.544884 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" event={"ID":"c194ac77-f017-48ed-916f-bd7425f9dcb9","Type":"ContainerDied","Data":"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b"} Nov 01 22:15:35.545016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.544923 5036 scope.go:122] "RemoveContainer" containerID="c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b" Nov 01 22:15:35.547165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.546805 5036 generic.go:356] "Generic (PLEG): container finished" podID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerID="c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b" exitCode=0 Nov 01 22:15:35.547165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.546925 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" event={"ID":"c194ac77-f017-48ed-916f-bd7425f9dcb9","Type":"ContainerDied","Data":"e881e97bbe630b84e61b629faa5ce0fa9f3ebd472a780ad1e36c4d93e312be64"} Nov 01 22:15:35.549346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.549253 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" event={"ID":"a4c18c69-1d1b-4912-a455-50423b6c61a3","Type":"ContainerStarted","Data":"67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d"} Nov 01 22:15:35.559574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.559502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" event={"ID":"8035e7b5-52af-4052-8301-85fedea4695b","Type":"ContainerStarted","Data":"cbff6f7a2a46fa1270fd2a75910e4eb37b58d03d367885b4316abb0ab8190030"} Nov 01 22:15:35.561504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.561485 5036 scope.go:122] "RemoveContainer" containerID="c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b" Nov 01 22:15:35.562182 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:35.562032 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b\": container with ID starting with c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b not found: ID does not exist" containerID="c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b" Nov 01 22:15:35.562182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.562118 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b"} err="failed to get container status \"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b\": rpc error: code = NotFound desc = could not find container \"c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b\": container with ID starting with c3787edd8598752e4ffb499e46e2ce598fa9cd38433fd4e49e670eb14eb6e80b not found: ID does not exist" Nov 01 22:15:35.573159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573136 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.573255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573162 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gxt\" (UniqueName: \"kubernetes.io/projected/c194ac77-f017-48ed-916f-bd7425f9dcb9-kube-api-access-g4gxt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.573255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573175 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194ac77-f017-48ed-916f-bd7425f9dcb9-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.573255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573184 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.573255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573194 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.573255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.573202 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194ac77-f017-48ed-916f-bd7425f9dcb9-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:35.587930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.587902 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:15:35.593157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:35.593133 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb"] Nov 01 22:15:36.156962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.156878 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2760d/test-model-f2760d-kserve-74f8b46877-gcmrb" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:36.318120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.318079 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:36.446177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.446100 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" path="/var/lib/kubelet/pods/c194ac77-f017-48ed-916f-bd7425f9dcb9/volumes" Nov 01 22:15:36.573425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.573381 5036 generic.go:356] "Generic (PLEG): container finished" podID="a4c18c69-1d1b-4912-a455-50423b6c61a3" containerID="920a517732a13ae43ebc30680f3e48ab2ec13c82224ec0427e748e947d26e4cc" exitCode=0 Nov 01 22:15:36.573664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.573497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" event={"ID":"a4c18c69-1d1b-4912-a455-50423b6c61a3","Type":"ContainerDied","Data":"920a517732a13ae43ebc30680f3e48ab2ec13c82224ec0427e748e947d26e4cc"} Nov 01 22:15:36.575471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.575429 5036 generic.go:356] "Generic (PLEG): container finished" podID="8035e7b5-52af-4052-8301-85fedea4695b" containerID="d1c2e81750090c49c2b4140faca4b3b2e111f2f0109e4dfb8cb48440a25eddd5" exitCode=0 Nov 01 22:15:36.575599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.575476 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" event={"ID":"8035e7b5-52af-4052-8301-85fedea4695b","Type":"ContainerDied","Data":"d1c2e81750090c49c2b4140faca4b3b2e111f2f0109e4dfb8cb48440a25eddd5"} Nov 01 22:15:36.785385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.785354 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:36.902637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.902608 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\" (UniqueName: \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\") pod \"60e9064d-fd73-4909-925c-70aa883720ac\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " Nov 01 22:15:36.902833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.902812 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\") pod \"60e9064d-fd73-4909-925c-70aa883720ac\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " Nov 01 22:15:36.902905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.902890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\") pod \"60e9064d-fd73-4909-925c-70aa883720ac\" (UID: \"60e9064d-fd73-4909-925c-70aa883720ac\") " Nov 01 22:15:36.903246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.903202 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca" pod "60e9064d-fd73-4909-925c-70aa883720ac" (UID: "60e9064d-fd73-4909-925c-70aa883720ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:36.903681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.903659 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:36.904961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.904929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55" pod "60e9064d-fd73-4909-925c-70aa883720ac" (UID: "60e9064d-fd73-4909-925c-70aa883720ac"). InnerVolumeSpecName "kube-api-access-f4d55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:36.905578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:36.905427 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca" pod "60e9064d-fd73-4909-925c-70aa883720ac" (UID: "60e9064d-fd73-4909-925c-70aa883720ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:37.005606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.005507 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4d55\" (UniqueName: \"kubernetes.io/projected/60e9064d-fd73-4909-925c-70aa883720ac-kube-api-access-f4d55\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:37.005606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.005545 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e9064d-fd73-4909-925c-70aa883720ac-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:37.159548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.159490 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:37.589855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.589822 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" Nov 01 22:15:37.589855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.589827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh" event={"ID":"60e9064d-fd73-4909-925c-70aa883720ac","Type":"ContainerDied","Data":"3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4"} Nov 01 22:15:37.590126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.589888 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3380ee0da2b169c81a74b0985c1e7cf569ffab77d3f4b36adb8857b4917ca1b4" Nov 01 22:15:37.861094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.854177 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:37.914392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.903364 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:37.926063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.921405 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:37.940278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940236 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\") pod \"8035e7b5-52af-4052-8301-85fedea4695b\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " Nov 01 22:15:37.940479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940338 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\" (UniqueName: \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\") pod \"a4c18c69-1d1b-4912-a455-50423b6c61a3\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " Nov 01 22:15:37.940573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940542 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\") pod \"8035e7b5-52af-4052-8301-85fedea4695b\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " Nov 01 22:15:37.940621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\") pod \"a4c18c69-1d1b-4912-a455-50423b6c61a3\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " Nov 01 22:15:37.940820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940805 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\" (UniqueName: \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\") pod \"8035e7b5-52af-4052-8301-85fedea4695b\" (UID: \"8035e7b5-52af-4052-8301-85fedea4695b\") " Nov 01 22:15:37.940887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940847 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\") pod \"a4c18c69-1d1b-4912-a455-50423b6c61a3\" (UID: \"a4c18c69-1d1b-4912-a455-50423b6c61a3\") " Nov 01 22:15:37.940887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.940864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca" pod "8035e7b5-52af-4052-8301-85fedea4695b" (UID: "8035e7b5-52af-4052-8301-85fedea4695b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:37.941908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.941879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:37.944065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.944028 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca" pod "a4c18c69-1d1b-4912-a455-50423b6c61a3" (UID: "a4c18c69-1d1b-4912-a455-50423b6c61a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:37.945963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.945939 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca" pod "8035e7b5-52af-4052-8301-85fedea4695b" (UID: "8035e7b5-52af-4052-8301-85fedea4695b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:37.947200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.946041 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca" pod "a4c18c69-1d1b-4912-a455-50423b6c61a3" (UID: "a4c18c69-1d1b-4912-a455-50423b6c61a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:37.947200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.946072 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm" pod "a4c18c69-1d1b-4912-a455-50423b6c61a3" (UID: "a4c18c69-1d1b-4912-a455-50423b6c61a3"). InnerVolumeSpecName "kube-api-access-vdzzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:37.947200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:37.946978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t" pod "8035e7b5-52af-4052-8301-85fedea4695b" (UID: "8035e7b5-52af-4052-8301-85fedea4695b"). InnerVolumeSpecName "kube-api-access-tcf7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:38.043751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.043715 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzzm\" (UniqueName: \"kubernetes.io/projected/a4c18c69-1d1b-4912-a455-50423b6c61a3-kube-api-access-vdzzm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:38.043751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.043748 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8035e7b5-52af-4052-8301-85fedea4695b-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:38.043751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.043759 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:38.043981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.043769 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf7t\" (UniqueName: \"kubernetes.io/projected/8035e7b5-52af-4052-8301-85fedea4695b-kube-api-access-tcf7t\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:38.043981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.043778 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c18c69-1d1b-4912-a455-50423b6c61a3-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:38.606296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.606268 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" Nov 01 22:15:38.606810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.606688 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb" event={"ID":"a4c18c69-1d1b-4912-a455-50423b6c61a3","Type":"ContainerDied","Data":"67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d"} Nov 01 22:15:38.606810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.606753 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67bdf586553e89e390b65305a6b409338c6973ebccb90b80f8abe6f44537812d" Nov 01 22:15:38.608041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.608017 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" Nov 01 22:15:38.608203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.608042 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7" event={"ID":"8035e7b5-52af-4052-8301-85fedea4695b","Type":"ContainerDied","Data":"cbff6f7a2a46fa1270fd2a75910e4eb37b58d03d367885b4316abb0ab8190030"} Nov 01 22:15:38.608203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.608093 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbff6f7a2a46fa1270fd2a75910e4eb37b58d03d367885b4316abb0ab8190030" Nov 01 22:15:38.715245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:38.715147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:39.444478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:39.444431 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:40.433704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:40.433664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:41.376068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:41.376032 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:42.522467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.507544 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:15:42.610607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.610553 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:42.612907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.612834 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.233:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:42.672466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.671217 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": context deadline exceeded" Nov 01 22:15:42.788544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.783947 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:42.904518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.896534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:42.926406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.920067 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:42.981475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:42.979543 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:15:43.906877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:43.906832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:44.042465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:44.041744 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:44.164000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:44.163905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:44.752129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:44.752086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:44.984806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:44.961718 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:45.421210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:45.421172 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:46.208438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.208354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:46.338389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.338316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:46.372803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.372741 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb"] Nov 01 22:15:46.380950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.380919 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d0a5de-zvnxb"] Nov 01 22:15:46.462519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.462360 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c18c69-1d1b-4912-a455-50423b6c61a3" path="/var/lib/kubelet/pods/a4c18c69-1d1b-4912-a455-50423b6c61a3/volumes" Nov 01 22:15:46.499474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.497917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:46.926468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:46.906396 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:47.060687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.060644 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:47.133098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.133065 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7"] Nov 01 22:15:47.141369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.141338 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2760d-ppjx7"] Nov 01 22:15:47.192283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.192192 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:47.513907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.513821 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" containerID="cri-o://8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3" gracePeriod=45 Nov 01 22:15:47.703033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.703002 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:47.852541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.852511 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:15:47.936025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.935994 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.936257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.936152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.936257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.936186 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\" (UniqueName: \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.936397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.936278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.936658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.936636 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.936767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.936717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\") pod \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\" (UID: \"34e4fc1d-d15d-42d6-a556-3ec303761bd2\") " Nov 01 22:15:47.937600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.937549 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:47.937764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.937604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:47.937764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.937712 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:47.938659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.938631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:47.939478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.939433 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "kube-api-access-zdxqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:47.939546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:47.939438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs" pod "34e4fc1d-d15d-42d6-a556-3ec303761bd2" (UID: "34e4fc1d-d15d-42d6-a556-3ec303761bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:48.039537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039499 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.039537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039537 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.039762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039552 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxqm\" (UniqueName: \"kubernetes.io/projected/34e4fc1d-d15d-42d6-a556-3ec303761bd2-kube-api-access-zdxqm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.039762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039568 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.039762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039584 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.039762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.039597 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e4fc1d-d15d-42d6-a556-3ec303761bd2-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:48.061426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.061392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:48.200198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.200102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:48.344252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.344212 5036 generic.go:356] "Generic (PLEG): container finished" podID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerID="8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3" exitCode=0 Nov 01 22:15:48.344562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.344290 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" Nov 01 22:15:48.344562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.344372 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" event={"ID":"34e4fc1d-d15d-42d6-a556-3ec303761bd2","Type":"ContainerDied","Data":"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3"} Nov 01 22:15:48.344562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.344412 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz" event={"ID":"34e4fc1d-d15d-42d6-a556-3ec303761bd2","Type":"ContainerDied","Data":"6c2af65647304432fa0c877aadb8f4e14936c72ade3354c76c6a83f0ba032ccf"} Nov 01 22:15:48.344562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.344514 5036 scope.go:122] "RemoveContainer" containerID="8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3" Nov 01 22:15:48.350317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.350284 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:48.357897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.357876 5036 scope.go:122] "RemoveContainer" containerID="8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3" Nov 01 22:15:48.358177 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:48.358157 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3\": container with ID starting with 8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3 not found: ID does not exist" containerID="8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3" Nov 01 22:15:48.358263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.358184 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3"} err="failed to get container status \"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3\": rpc error: code = NotFound desc = could not find container \"8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3\": container with ID starting with 8bf13660ba9cae2e29208390a7d6e66e5a87bf94e28cc780b22bd234cf7fd6d3 not found: ID does not exist" Nov 01 22:15:48.371660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.371628 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:15:48.376792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.376769 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-17957f/test-model-17957f-kserve-855ddd64d5-f72qz"] Nov 01 22:15:48.441662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.441632 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" path="/var/lib/kubelet/pods/34e4fc1d-d15d-42d6-a556-3ec303761bd2/volumes" Nov 01 22:15:48.442252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.442236 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8035e7b5-52af-4052-8301-85fedea4695b" path="/var/lib/kubelet/pods/8035e7b5-52af-4052-8301-85fedea4695b/volumes" Nov 01 22:15:48.702475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:48.698613 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" containerID="cri-o://e0340cfb2998d47aaa88f9ab3ba5837df6d6ede7ee55ea97164ce88220578177" gracePeriod=45 Nov 01 22:15:48.958814 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:48.958734 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:15:49.057466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.056464 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:49.295313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.294958 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": dial tcp 10.132.0.220:8000: connect: connection refused" Nov 01 22:15:49.360262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.360231 5036 generic.go:356] "Generic (PLEG): container finished" podID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerID="e0340cfb2998d47aaa88f9ab3ba5837df6d6ede7ee55ea97164ce88220578177" exitCode=0 Nov 01 22:15:49.360432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.360417 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" event={"ID":"b3b2c441-fb50-46bd-ae3b-a0f346b033e2","Type":"ContainerDied","Data":"e0340cfb2998d47aaa88f9ab3ba5837df6d6ede7ee55ea97164ce88220578177"} Nov 01 22:15:49.439793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.439758 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:15:49.474708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.474616 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:49.602556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.602387 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" containerID="cri-o://b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093" gracePeriod=45 Nov 01 22:15:49.843580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.843545 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h"] Nov 01 22:15:49.844895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844786 5036 cpu_manager.go:414] "containerMap: removing container" podUID="60e9064d-fd73-4909-925c-70aa883720ac" containerName="revoke-keys" Nov 01 22:15:49.844895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844812 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e9064d-fd73-4909-925c-70aa883720ac" containerName="revoke-keys" Nov 01 22:15:49.844895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844834 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" Nov 01 22:15:49.844895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844842 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844898 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a4c18c69-1d1b-4912-a455-50423b6c61a3" containerName="revoke-keys" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844909 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c18c69-1d1b-4912-a455-50423b6c61a3" containerName="revoke-keys" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844944 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8035e7b5-52af-4052-8301-85fedea4695b" containerName="revoke-keys" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844951 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8035e7b5-52af-4052-8301-85fedea4695b" containerName="revoke-keys" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844963 5036 cpu_manager.go:414] "containerMap: removing container" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" Nov 01 22:15:49.845185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.844970 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" Nov 01 22:15:49.845482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.845204 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e9064d-fd73-4909-925c-70aa883720ac" containerName="revoke-keys" Nov 01 22:15:49.845482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.845219 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c194ac77-f017-48ed-916f-bd7425f9dcb9" containerName="main" Nov 01 22:15:49.845482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.845236 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c18c69-1d1b-4912-a455-50423b6c61a3" containerName="revoke-keys" Nov 01 22:15:49.845482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.845252 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8035e7b5-52af-4052-8301-85fedea4695b" containerName="revoke-keys" Nov 01 22:15:49.845482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.845262 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e4fc1d-d15d-42d6-a556-3ec303761bd2" containerName="main" Nov 01 22:15:49.852641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.852588 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:49.855719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.855675 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h"] Nov 01 22:15:49.859966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.859943 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:49.860077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.860056 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:15:49.970687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.970653 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:49.970853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.970834 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sbn\" (UniqueName: \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:49.971146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.971122 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:49.984807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:49.984681 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:15:50.073343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.073296 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.073569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.073546 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sbn\" (UniqueName: \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.073872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.073848 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.074280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.074252 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.077120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.077078 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.108254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.107974 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sbn\" (UniqueName: \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\") pod \"maas-api-revoke-keys-e2e-inf-a-17957f-srt8h\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.158625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.158587 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:15:50.174535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.174503 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.174776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.174587 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.174776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.174764 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\" (UniqueName: \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.174926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.174880 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.174986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.174948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.175088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.175073 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\") pod \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\" (UID: \"b3b2c441-fb50-46bd-ae3b-a0f346b033e2\") " Nov 01 22:15:50.175231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.175188 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.175515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.175345 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.175662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.175624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.176060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.175997 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.176060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.176039 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.176060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.176061 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.179712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.179060 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.179712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.179438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "kube-api-access-99pdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:50.180202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.180144 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs" pod "b3b2c441-fb50-46bd-ae3b-a0f346b033e2" (UID: "b3b2c441-fb50-46bd-ae3b-a0f346b033e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:50.277370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277390 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277633 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\" (UniqueName: \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277739 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.277974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\") pod \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\" (UID: \"a3a2eb43-2e70-4f89-8aea-ad68d029d500\") " Nov 01 22:15:50.277974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.277930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.278284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278239 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.278823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278795 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.278823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278825 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.278989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278839 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.278989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278852 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.278989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278866 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99pdj\" (UniqueName: \"kubernetes.io/projected/b3b2c441-fb50-46bd-ae3b-a0f346b033e2-kube-api-access-99pdj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.278989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.278879 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.280000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.279972 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:50.280110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.280044 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:50.280179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.280111 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "kube-api-access-rhvqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:50.280351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.280315 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs" pod "a3a2eb43-2e70-4f89-8aea-ad68d029d500" (UID: "a3a2eb43-2e70-4f89-8aea-ad68d029d500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:50.380801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.380727 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a2eb43-2e70-4f89-8aea-ad68d029d500-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.380801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.380756 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvqq\" (UniqueName: \"kubernetes.io/projected/a3a2eb43-2e70-4f89-8aea-ad68d029d500-kube-api-access-rhvqq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.380801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.380786 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a2eb43-2e70-4f89-8aea-ad68d029d500-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:50.383178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.382997 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" Nov 01 22:15:50.383178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.383056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b" event={"ID":"b3b2c441-fb50-46bd-ae3b-a0f346b033e2","Type":"ContainerDied","Data":"248affcba3d308578fe0d67352723f3a0b27a226e7ceac8be55a773fee9a1922"} Nov 01 22:15:50.383178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.383097 5036 scope.go:122] "RemoveContainer" containerID="e0340cfb2998d47aaa88f9ab3ba5837df6d6ede7ee55ea97164ce88220578177" Nov 01 22:15:50.386589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.386543 5036 generic.go:356] "Generic (PLEG): container finished" podID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerID="b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093" exitCode=0 Nov 01 22:15:50.386716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.386700 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" Nov 01 22:15:50.390623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.390598 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" event={"ID":"a3a2eb43-2e70-4f89-8aea-ad68d029d500","Type":"ContainerDied","Data":"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093"} Nov 01 22:15:50.390737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.390639 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd" event={"ID":"a3a2eb43-2e70-4f89-8aea-ad68d029d500","Type":"ContainerDied","Data":"588429239aa18447369d51ee49f72accb06c444fcef747117527f3701c94bf7a"} Nov 01 22:15:50.405225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.405072 5036 scope.go:122] "RemoveContainer" containerID="b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093" Nov 01 22:15:50.426197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.426148 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:15:50.428049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.428027 5036 scope.go:122] "RemoveContainer" containerID="b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093" Nov 01 22:15:50.428402 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:50.428369 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093\": container with ID starting with b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093 not found: ID does not exist" containerID="b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093" Nov 01 22:15:50.428505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.428409 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093"} err="failed to get container status \"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093\": rpc error: code = NotFound desc = could not find container \"b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093\": container with ID starting with b2757db9a955ff9dcb2f4f034660092862d08d29ce27202b4e7f8aeba7903093 not found: ID does not exist" Nov 01 22:15:50.444387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.444356 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9d8b5d/test-model-9d8b5d-kserve-59cbd98bb7-x4pkd"] Nov 01 22:15:50.444509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.444401 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:15:50.446251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.446231 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8483c2/test-model-8483c2-kserve-b9f44f459-r6w8b"] Nov 01 22:15:50.458841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.458799 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:50.577652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:50.577614 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h"] Nov 01 22:15:51.398993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:51.398951 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:51.401764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:51.401726 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" event={"ID":"a416b131-3b53-41b4-9c3d-a1ffeb808965","Type":"ContainerStarted","Data":"ed2a6862eac116026553b3efe6c6a711bacd4f0f3148f26053278a1a0075386e"} Nov 01 22:15:51.401928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:51.401818 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" event={"ID":"a416b131-3b53-41b4-9c3d-a1ffeb808965","Type":"ContainerStarted","Data":"e50d59d79a5e4c97d3186178c66ab976fac7df42f084b229791d6ad1c5508c19"} Nov 01 22:15:51.470937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:51.470871 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" podStartSLOduration=2.470851984 podStartE2EDuration="2.470851984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:51.440468852 +0000 UTC m=+1905.133449336" watchObservedRunningTime="2026-11-01 22:15:51.470851984 +0000 UTC m=+1905.163832470" Nov 01 22:15:52.002742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.002655 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:15:52.003848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.003826 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" Nov 01 22:15:52.003919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.003852 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" Nov 01 22:15:52.003967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.003934 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" Nov 01 22:15:52.003967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.003944 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" Nov 01 22:15:52.004260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.004222 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" containerName="main" Nov 01 22:15:52.004260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.004241 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" containerName="main" Nov 01 22:15:52.017747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.017718 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:15:52.017892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.017832 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.028508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.025847 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7e2df\"/\"kube-root-ca.crt\"" Nov 01 22:15:52.028508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.026721 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7e2df\"/\"default-dockercfg-5gr4t\"" Nov 01 22:15:52.028508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.026830 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7e2df\"/\"test-model-e7e2df-kserve-self-signed-certs\"" Nov 01 22:15:52.028508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.026888 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7e2df\"/\"openshift-service-ca.crt\"" Nov 01 22:15:52.052526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.051068 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.074886 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.075068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.075245 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqtft\" (UniqueName: \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.075426 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.075477 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.077485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.075544 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177247 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177524 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177598 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177661 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtft\" (UniqueName: \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177700 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177816 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.177905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.177854 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.178084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.178059 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.178267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.178233 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.180191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.180160 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.180862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.180837 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.190132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.190105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtft\" (UniqueName: \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\") pod \"test-model-e7e2df-kserve-5ddd9df877-xbzn8\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.191137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.191109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:52.341467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.340489 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:52.545798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.524737 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a2eb43-2e70-4f89-8aea-ad68d029d500" path="/var/lib/kubelet/pods/a3a2eb43-2e70-4f89-8aea-ad68d029d500/volumes" Nov 01 22:15:52.545798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.541133 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b2c441-fb50-46bd-ae3b-a0f346b033e2" path="/var/lib/kubelet/pods/b3b2c441-fb50-46bd-ae3b-a0f346b033e2/volumes" Nov 01 22:15:52.559530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.559390 5036 generic.go:356] "Generic (PLEG): container finished" podID="a416b131-3b53-41b4-9c3d-a1ffeb808965" containerID="ed2a6862eac116026553b3efe6c6a711bacd4f0f3148f26053278a1a0075386e" exitCode=0 Nov 01 22:15:52.559689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.559557 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" event={"ID":"a416b131-3b53-41b4-9c3d-a1ffeb808965","Type":"ContainerDied","Data":"ed2a6862eac116026553b3efe6c6a711bacd4f0f3148f26053278a1a0075386e"} Nov 01 22:15:52.851743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.851709 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:15:52.857330 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:52.857301 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28fca3fe_885b_4442_aed8_be0edba2a2b2.slice/crio-93e2a5e44290e952632b9774ae9104bc9e2ac9d4df1f6a6bd41dd5ab765f180e WatchSource:0}: Error finding container 93e2a5e44290e952632b9774ae9104bc9e2ac9d4df1f6a6bd41dd5ab765f180e: Status 404 returned error can't find the container with id 93e2a5e44290e952632b9774ae9104bc9e2ac9d4df1f6a6bd41dd5ab765f180e Nov 01 22:15:52.948469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:52.936905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:53.571204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.571159 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" event={"ID":"28fca3fe-885b-4442-aed8-be0edba2a2b2","Type":"ContainerStarted","Data":"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950"} Nov 01 22:15:53.571649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.571210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" event={"ID":"28fca3fe-885b-4442-aed8-be0edba2a2b2","Type":"ContainerStarted","Data":"93e2a5e44290e952632b9774ae9104bc9e2ac9d4df1f6a6bd41dd5ab765f180e"} Nov 01 22:15:53.594769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.594706 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podStartSLOduration=2.594691649 podStartE2EDuration="2.594691649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:53.590310829 +0000 UTC m=+1907.283291317" watchObservedRunningTime="2026-11-01 22:15:53.594691649 +0000 UTC m=+1907.287672135" Nov 01 22:15:53.666025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.665986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:53.785088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.785056 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:53.944238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.944160 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\" (UniqueName: \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\") pod \"a416b131-3b53-41b4-9c3d-a1ffeb808965\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " Nov 01 22:15:53.944421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.944283 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\") pod \"a416b131-3b53-41b4-9c3d-a1ffeb808965\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " Nov 01 22:15:53.944421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.944312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\") pod \"a416b131-3b53-41b4-9c3d-a1ffeb808965\" (UID: \"a416b131-3b53-41b4-9c3d-a1ffeb808965\") " Nov 01 22:15:53.945021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.944985 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca" pod "a416b131-3b53-41b4-9c3d-a1ffeb808965" (UID: "a416b131-3b53-41b4-9c3d-a1ffeb808965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:53.945313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.945291 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:53.946736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.946709 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn" pod "a416b131-3b53-41b4-9c3d-a1ffeb808965" (UID: "a416b131-3b53-41b4-9c3d-a1ffeb808965"). InnerVolumeSpecName "kube-api-access-b8sbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:53.947237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:53.947209 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca" pod "a416b131-3b53-41b4-9c3d-a1ffeb808965" (UID: "a416b131-3b53-41b4-9c3d-a1ffeb808965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:54.047149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.047113 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a416b131-3b53-41b4-9c3d-a1ffeb808965-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:54.047149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.047150 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8sbn\" (UniqueName: \"kubernetes.io/projected/a416b131-3b53-41b4-9c3d-a1ffeb808965-kube-api-access-b8sbn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:54.071133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.071097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:54.206117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.206030 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:54.341907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.341875 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:15:54.346084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.346047 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:15:54.346743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.346688 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" containerID="cri-o://c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5" gracePeriod=30 Nov 01 22:15:54.398462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.396842 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:54.710927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.710888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:54.718286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.718258 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" Nov 01 22:15:54.735668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.718499 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h" event={"ID":"a416b131-3b53-41b4-9c3d-a1ffeb808965","Type":"ContainerDied","Data":"e50d59d79a5e4c97d3186178c66ab976fac7df42f084b229791d6ad1c5508c19"} Nov 01 22:15:54.736927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.736901 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50d59d79a5e4c97d3186178c66ab976fac7df42f084b229791d6ad1c5508c19" Nov 01 22:15:54.828106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.828070 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq"] Nov 01 22:15:54.829481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.829434 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a416b131-3b53-41b4-9c3d-a1ffeb808965" containerName="revoke-keys" Nov 01 22:15:54.829481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.829473 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416b131-3b53-41b4-9c3d-a1ffeb808965" containerName="revoke-keys" Nov 01 22:15:54.882561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.861043 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:54.882561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.872467 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416b131-3b53-41b4-9c3d-a1ffeb808965" containerName="revoke-keys" Nov 01 22:15:54.905496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.902650 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq"] Nov 01 22:15:54.905496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.902752 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:54.928463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.926386 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:15:54.928463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.926472 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:54.979519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.977411 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdvv\" (UniqueName: \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:54.980089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.980061 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:54.984484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:54.980436 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.009541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.002713 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:15:55.009541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.003069 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" containerID="cri-o://21f4f1f947365569db14c22d4ab809f0400fa68378860e7c3fb59f3b16d0caa7" gracePeriod=30 Nov 01 22:15:55.016576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.016534 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:15:55.016796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.016769 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" containerID="cri-o://96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb" gracePeriod=30 Nov 01 22:15:55.040086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.040053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:55.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.094978 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.095243 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.095807 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdvv\" (UniqueName: \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.131468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.128112 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.161480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.157400 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdvv\" (UniqueName: \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.161480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.157811 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.247897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.247574 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:55.515791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.515301 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6"] Nov 01 22:15:55.523807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.522119 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.530575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.530543 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6"] Nov 01 22:15:55.631555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.631515 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.631890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.631870 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.631959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.631915 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzv2t\" (UniqueName: \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.696331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.696296 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq"] Nov 01 22:15:55.696614 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:55.696566 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0436a9_938d_4fa1_9188_f1527b68b4bb.slice/crio-0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24 WatchSource:0}: Error finding container 0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24: Status 404 returned error can't find the container with id 0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24 Nov 01 22:15:55.729779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.729747 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" event={"ID":"fc0436a9-938d-4fa1-9188-f1527b68b4bb","Type":"ContainerStarted","Data":"0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24"} Nov 01 22:15:55.733121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.733098 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.733200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.733143 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzv2t\" (UniqueName: \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.733264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.733196 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.733898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.733867 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.735644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.735622 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.745343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.745321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzv2t\" (UniqueName: \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\") pod \"maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.855958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.855928 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:55.987176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:55.987141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:56.123059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.122972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:56.153816 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:15:56.153779 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd592b36b_33af_4307_83d5_1c5099e905a7.slice/crio-63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b WatchSource:0}: Error finding container 63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b: Status 404 returned error can't find the container with id 63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b Nov 01 22:15:56.157994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.157967 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6"] Nov 01 22:15:56.321010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.320932 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:56.742102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.741996 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" event={"ID":"fc0436a9-938d-4fa1-9188-f1527b68b4bb","Type":"ContainerStarted","Data":"e6101e5b9fd4530c56441a10b4d202c99bea9687b07c41230a970c4d2030e575"} Nov 01 22:15:56.743943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.743912 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" event={"ID":"d592b36b-33af-4307-83d5-1c5099e905a7","Type":"ContainerStarted","Data":"63b9d214bbebd4152ac8749e2bb971d442ab0d060da48b33e3951b8e2d6fa1dc"} Nov 01 22:15:56.744111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.743968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" event={"ID":"d592b36b-33af-4307-83d5-1c5099e905a7","Type":"ContainerStarted","Data":"63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b"} Nov 01 22:15:56.765017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.764963 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" podStartSLOduration=2.764950037 podStartE2EDuration="2.764950037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:56.759878011 +0000 UTC m=+1910.452858498" watchObservedRunningTime="2026-11-01 22:15:56.764950037 +0000 UTC m=+1910.457930552" Nov 01 22:15:56.786503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:56.786437 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" podStartSLOduration=1.786420341 podStartE2EDuration="1.786420341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:56.781009379 +0000 UTC m=+1910.473989870" watchObservedRunningTime="2026-11-01 22:15:56.786420341 +0000 UTC m=+1910.479400827" Nov 01 22:15:57.241321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:57.241247 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:57.755048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:57.755011 5036 generic.go:356] "Generic (PLEG): container finished" podID="fc0436a9-938d-4fa1-9188-f1527b68b4bb" containerID="e6101e5b9fd4530c56441a10b4d202c99bea9687b07c41230a970c4d2030e575" exitCode=0 Nov 01 22:15:57.755507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:57.755101 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" event={"ID":"fc0436a9-938d-4fa1-9188-f1527b68b4bb","Type":"ContainerDied","Data":"e6101e5b9fd4530c56441a10b4d202c99bea9687b07c41230a970c4d2030e575"} Nov 01 22:15:57.756984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:57.756958 5036 generic.go:356] "Generic (PLEG): container finished" podID="d592b36b-33af-4307-83d5-1c5099e905a7" containerID="63b9d214bbebd4152ac8749e2bb971d442ab0d060da48b33e3951b8e2d6fa1dc" exitCode=0 Nov 01 22:15:57.757112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:57.757050 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" event={"ID":"d592b36b-33af-4307-83d5-1c5099e905a7","Type":"ContainerDied","Data":"63b9d214bbebd4152ac8749e2bb971d442ab0d060da48b33e3951b8e2d6fa1dc"} Nov 01 22:15:58.048203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:58.048155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:58.846697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:58.846601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:59.065974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.065942 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:59.135403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.134249 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\" (UniqueName: \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\") pod \"d592b36b-33af-4307-83d5-1c5099e905a7\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " Nov 01 22:15:59.135403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.134313 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\") pod \"d592b36b-33af-4307-83d5-1c5099e905a7\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " Nov 01 22:15:59.135403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.134386 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\") pod \"d592b36b-33af-4307-83d5-1c5099e905a7\" (UID: \"d592b36b-33af-4307-83d5-1c5099e905a7\") " Nov 01 22:15:59.139582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.139544 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca" pod "d592b36b-33af-4307-83d5-1c5099e905a7" (UID: "d592b36b-33af-4307-83d5-1c5099e905a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:59.140597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.140568 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.141907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.141833 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:59.143000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.142975 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t" pod "d592b36b-33af-4307-83d5-1c5099e905a7" (UID: "d592b36b-33af-4307-83d5-1c5099e905a7"). InnerVolumeSpecName "kube-api-access-tzv2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:59.143967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.143931 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca" pod "d592b36b-33af-4307-83d5-1c5099e905a7" (UID: "d592b36b-33af-4307-83d5-1c5099e905a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:59.241911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.241868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\") pod \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " Nov 01 22:15:59.242086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.241992 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\") pod \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " Nov 01 22:15:59.242086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.242073 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\" (UniqueName: \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\") pod \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\" (UID: \"fc0436a9-938d-4fa1-9188-f1527b68b4bb\") " Nov 01 22:15:59.242411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.242375 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca" pod "fc0436a9-938d-4fa1-9188-f1527b68b4bb" (UID: "fc0436a9-938d-4fa1-9188-f1527b68b4bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:59.242983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.242956 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzv2t\" (UniqueName: \"kubernetes.io/projected/d592b36b-33af-4307-83d5-1c5099e905a7-kube-api-access-tzv2t\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.242983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.242977 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.243197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.242988 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d592b36b-33af-4307-83d5-1c5099e905a7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.244884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.244859 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv" pod "fc0436a9-938d-4fa1-9188-f1527b68b4bb" (UID: "fc0436a9-938d-4fa1-9188-f1527b68b4bb"). InnerVolumeSpecName "kube-api-access-tkdvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:59.245285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.245263 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca" pod "fc0436a9-938d-4fa1-9188-f1527b68b4bb" (UID: "fc0436a9-938d-4fa1-9188-f1527b68b4bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:59.344233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.344167 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdvv\" (UniqueName: \"kubernetes.io/projected/fc0436a9-938d-4fa1-9188-f1527b68b4bb-kube-api-access-tkdvv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.344233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.344196 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0436a9-938d-4fa1-9188-f1527b68b4bb-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.473958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.473918 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:59.741093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.741063 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:15:59.781310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.781284 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" Nov 01 22:15:59.781519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.781295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq" event={"ID":"fc0436a9-938d-4fa1-9188-f1527b68b4bb","Type":"ContainerDied","Data":"0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24"} Nov 01 22:15:59.781519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.781373 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aee10232d770b3785567aeae7631340516a642c7037751e6b0c424921498a24" Nov 01 22:15:59.783688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.783660 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" Nov 01 22:15:59.784514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.784478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6" event={"ID":"d592b36b-33af-4307-83d5-1c5099e905a7","Type":"ContainerDied","Data":"63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b"} Nov 01 22:15:59.784646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.784556 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63501addb43f210b10385b3c2bb925ac0d965f012aa6322ae15e7b5d716c450b" Nov 01 22:15:59.786893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.786871 5036 generic.go:356] "Generic (PLEG): container finished" podID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerID="c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5" exitCode=0 Nov 01 22:15:59.787075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.786951 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerDied","Data":"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5"} Nov 01 22:15:59.787185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.787094 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" event={"ID":"3687d95e-f5bf-4786-9b20-c9e1c15ab22a","Type":"ContainerDied","Data":"e9156e6dd6fc496755135260a1ec128b5b90e72698b980743b0ff5babcacbba8"} Nov 01 22:15:59.787185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.787128 5036 scope.go:122] "RemoveContainer" containerID="c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5" Nov 01 22:15:59.787185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.786985 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv" Nov 01 22:15:59.806128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.806109 5036 scope.go:122] "RemoveContainer" containerID="e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6" Nov 01 22:15:59.817111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.817084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:15:59.822211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.822187 5036 scope.go:122] "RemoveContainer" containerID="c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5" Nov 01 22:15:59.822663 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:59.822637 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5\": container with ID starting with c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5 not found: ID does not exist" containerID="c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5" Nov 01 22:15:59.822781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.822672 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5"} err="failed to get container status \"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5\": rpc error: code = NotFound desc = could not find container \"c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5\": container with ID starting with c7fe76d92c50444dad7990589947d020aa7cfee7c831ed0fa2fe4a8ba025adb5 not found: ID does not exist" Nov 01 22:15:59.822781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.822698 5036 scope.go:122] "RemoveContainer" containerID="e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6" Nov 01 22:15:59.823053 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:15:59.823026 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6\": container with ID starting with e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6 not found: ID does not exist" containerID="e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6" Nov 01 22:15:59.823131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.823077 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6"} err="failed to get container status \"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6\": rpc error: code = NotFound desc = could not find container \"e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6\": container with ID starting with e4fc5a5ab38b6000cf998c03f3c9ae4272bba9c32386ae2768137274682d4bd6 not found: ID does not exist" Nov 01 22:15:59.869917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.869844 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870013 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870102 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870209 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870251 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870284 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870292 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:59.870395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870348 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870541 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870554 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870615 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\") pod \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\" (UID: \"3687d95e-f5bf-4786-9b20-c9e1c15ab22a\") " Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870744 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:59.870842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.870828 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:59.871561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.871535 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.871675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.871565 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.871675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.871580 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.871675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.871595 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.871675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.871610 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.872576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.872549 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:59.872778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.872758 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:59.873153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.873127 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "kube-api-access-hbvjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:59.873242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.873222 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo" pod "3687d95e-f5bf-4786-9b20-c9e1c15ab22a" (UID: "3687d95e-f5bf-4786-9b20-c9e1c15ab22a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:59.973737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.973704 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.973737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.973731 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.973737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.973743 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvjx\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-kube-api-access-hbvjx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:59.973989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:15:59.973754 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3687d95e-f5bf-4786-9b20-c9e1c15ab22a-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.045226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.043941 5036 patch_prober.go:29] interesting pod/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:16:00.045226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.044000 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 01 22:16:00.104930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.104818 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:00.201675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.201072 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:16:00.292617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.292582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:00.317519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.301712 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d0a5de-openshift-default-79f4bc7977-l9bjv"] Nov 01 22:16:00.544545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.541235 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:00.600709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.600658 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" path="/var/lib/kubelet/pods/3687d95e-f5bf-4786-9b20-c9e1c15ab22a/volumes" Nov 01 22:16:00.801303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.801225 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:16:00.804503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.804234 5036 generic.go:356] "Generic (PLEG): container finished" podID="4417b012-afea-41c9-8046-35f17e09053e" containerID="96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb" exitCode=0 Nov 01 22:16:00.804503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.804314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerDied","Data":"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb"} Nov 01 22:16:00.804503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.804360 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" event={"ID":"4417b012-afea-41c9-8046-35f17e09053e","Type":"ContainerDied","Data":"e81de78f090dadadcbd8ae2cabd00cf250cad3aa0cc007ee4818c843d70c0910"} Nov 01 22:16:00.804503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.804387 5036 scope.go:122] "RemoveContainer" containerID="96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb" Nov 01 22:16:00.804892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.804571 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt" Nov 01 22:16:00.810315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.810285 5036 generic.go:356] "Generic (PLEG): container finished" podID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerID="21f4f1f947365569db14c22d4ab809f0400fa68378860e7c3fb59f3b16d0caa7" exitCode=0 Nov 01 22:16:00.810436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.810359 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" event={"ID":"e5279c42-55d9-4297-8279-fb4b2d65e4e0","Type":"ContainerDied","Data":"21f4f1f947365569db14c22d4ab809f0400fa68378860e7c3fb59f3b16d0caa7"} Nov 01 22:16:00.818564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.818537 5036 scope.go:122] "RemoveContainer" containerID="c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c" Nov 01 22:16:00.833021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.832939 5036 scope.go:122] "RemoveContainer" containerID="96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb" Nov 01 22:16:00.833300 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:00.833279 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb\": container with ID starting with 96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb not found: ID does not exist" containerID="96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb" Nov 01 22:16:00.833429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.833406 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb"} err="failed to get container status \"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb\": rpc error: code = NotFound desc = could not find container \"96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb\": container with ID starting with 96f0d985a1fd19579729b5cde179498191db0e5d306850f26974626b558d77fb not found: ID does not exist" Nov 01 22:16:00.833599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.833583 5036 scope.go:122] "RemoveContainer" containerID="c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c" Nov 01 22:16:00.834043 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:00.833952 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c\": container with ID starting with c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c not found: ID does not exist" containerID="c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c" Nov 01 22:16:00.834043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.833986 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c"} err="failed to get container status \"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c\": rpc error: code = NotFound desc = could not find container \"c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c\": container with ID starting with c30ffe60ebf0a2676560e6256de2784b84d7f7797536299de929dfdf98c8376c not found: ID does not exist" Nov 01 22:16:00.850177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850153 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850221 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850433 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850527 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850558 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.850900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850832 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.851017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\") pod \"4417b012-afea-41c9-8046-35f17e09053e\" (UID: \"4417b012-afea-41c9-8046-35f17e09053e\") " Nov 01 22:16:00.851017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.850934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.851132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851056 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:00.851132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851080 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.851578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851554 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.851799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851772 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4417b012-afea-41c9-8046-35f17e09053e-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.851914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851805 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.851914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851821 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.851914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.851838 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.853502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.853467 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:00.854241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.854209 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.854381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.854361 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "kube-api-access-z2x28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:00.854766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.854740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo" pod "4417b012-afea-41c9-8046-35f17e09053e" (UID: "4417b012-afea-41c9-8046-35f17e09053e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:00.862192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.862169 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:16:00.952870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.952842 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.952898 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.952949 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.952971 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953047 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953117 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953250 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953254 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953267 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.953374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953284 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953384 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\") pod \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\" (UID: \"e5279c42-55d9-4297-8279-fb4b2d65e4e0\") " Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953753 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.953780 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954264 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954283 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954292 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954300 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954309 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954317 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954326 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4417b012-afea-41c9-8046-35f17e09053e-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954336 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2x28\" (UniqueName: \"kubernetes.io/projected/4417b012-afea-41c9-8046-35f17e09053e-kube-api-access-z2x28\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954345 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4417b012-afea-41c9-8046-35f17e09053e-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.954358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.954353 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:00.955654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.955631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:00.955841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.955824 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:00.956109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.956079 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "kube-api-access-qrrqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:00.956873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:00.956850 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo" pod "e5279c42-55d9-4297-8279-fb4b2d65e4e0" (UID: "e5279c42-55d9-4297-8279-fb4b2d65e4e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:01.056141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.056063 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:01.056141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.056102 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrqb\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-kube-api-access-qrrqb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:01.056141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.056120 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:01.056141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.056133 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5279c42-55d9-4297-8279-fb4b2d65e4e0-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:01.139547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.139514 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:16:01.144530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.144493 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-s7dbt"] Nov 01 22:16:01.552924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.552886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:01.823020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.822933 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" Nov 01 22:16:01.823020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.822970 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c" event={"ID":"e5279c42-55d9-4297-8279-fb4b2d65e4e0","Type":"ContainerDied","Data":"e7b4d65dfed447c8b2e3f3ceb02bfd59afc44eb85be987d5257378507329a9e9"} Nov 01 22:16:01.823241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.823037 5036 scope.go:122] "RemoveContainer" containerID="21f4f1f947365569db14c22d4ab809f0400fa68378860e7c3fb59f3b16d0caa7" Nov 01 22:16:01.853398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.853365 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:16:01.858410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:01.858388 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f2760d-openshift-default-845f578cdf-qwc8c"] Nov 01 22:16:02.242257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.242158 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:02.341278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.341243 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:16:02.354586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.354559 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:16:02.393620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.393578 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:16:02.394946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.394912 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.394946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.394946 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.394969 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.394978 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395001 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395009 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395026 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395034 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395059 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395066 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395089 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d592b36b-33af-4307-83d5-1c5099e905a7" containerName="revoke-keys" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395097 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d592b36b-33af-4307-83d5-1c5099e905a7" containerName="revoke-keys" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395147 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fc0436a9-938d-4fa1-9188-f1527b68b4bb" containerName="revoke-keys" Nov 01 22:16:02.395161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395156 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0436a9-938d-4fa1-9188-f1527b68b4bb" containerName="revoke-keys" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395351 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0436a9-938d-4fa1-9188-f1527b68b4bb" containerName="revoke-keys" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395369 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395384 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" containerName="istio-proxy" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395398 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4417b012-afea-41c9-8046-35f17e09053e" containerName="istio-proxy" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395408 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395421 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3687d95e-f5bf-4786-9b20-c9e1c15ab22a" containerName="istio-proxy" Nov 01 22:16:02.395826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.395432 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d592b36b-33af-4307-83d5-1c5099e905a7" containerName="revoke-keys" Nov 01 22:16:02.403243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.403220 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.415058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.411126 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6b4c49\"/\"default-dockercfg-sztxq\"" Nov 01 22:16:02.415058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.411883 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6b4c49\"/\"openshift-service-ca.crt\"" Nov 01 22:16:02.415058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.411951 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6b4c49\"/\"kube-root-ca.crt\"" Nov 01 22:16:02.415058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.412054 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6b4c49\"/\"test-model-6b4c49-kserve-self-signed-certs\"" Nov 01 22:16:02.415058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.412092 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:16:02.443242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.443189 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4417b012-afea-41c9-8046-35f17e09053e" path="/var/lib/kubelet/pods/4417b012-afea-41c9-8046-35f17e09053e/volumes" Nov 01 22:16:02.444407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.444009 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5279c42-55d9-4297-8279-fb4b2d65e4e0" path="/var/lib/kubelet/pods/e5279c42-55d9-4297-8279-fb4b2d65e4e0/volumes" Nov 01 22:16:02.447844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.447811 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:02.485341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485314 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.485496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485395 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgpz\" (UniqueName: \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.485707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485680 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.485792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.485792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485777 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.485929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.485840 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.591752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.591844 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.591873 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.591901 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.592007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.592084 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgpz\" (UniqueName: \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.603467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.592943 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.615494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.605378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.615494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.605653 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.634341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.634307 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.650476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.644614 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.650476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.645298 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgpz\" (UniqueName: \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\") pod \"test-model-6b4c49-kserve-6594bc78f5-74mkt\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.740772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.740498 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:02.955167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:02.955133 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:16:03.124243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.124160 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:03.129473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.127140 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:16:03.920138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.920078 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" event={"ID":"7ff32747-47d4-4e81-8606-1eec5d959069","Type":"ContainerStarted","Data":"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7"} Nov 01 22:16:03.920935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.920883 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" event={"ID":"7ff32747-47d4-4e81-8606-1eec5d959069","Type":"ContainerStarted","Data":"422914fc11e1bd67b9f0ecfd51d9083ca91672d2d9d769ed75746bcc5d8bfaf7"} Nov 01 22:16:03.940107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.940047 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podStartSLOduration=1.940030302 podStartE2EDuration="1.940030302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:03.939861754 +0000 UTC m=+1917.632842241" watchObservedRunningTime="2026-11-01 22:16:03.940030302 +0000 UTC m=+1917.633010791" Nov 01 22:16:03.941173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:03.941125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:04.487731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.487691 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" containerID="cri-o://436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b" gracePeriod=45 Nov 01 22:16:04.740914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.740823 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:04.812636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.812609 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:16:04.873156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.873117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:04.956370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956187 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.956986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956413 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.956986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956485 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.956986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956541 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\" (UniqueName: \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.956986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956575 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.956986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.956641 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\") pod \"0b3d355a-afae-4e36-84db-c10434e04638\" (UID: \"0b3d355a-afae-4e36-84db-c10434e04638\") " Nov 01 22:16:04.958746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.958567 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:04.958746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.958648 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:04.959100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.958988 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:04.959640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.959617 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:04.959761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.959646 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:04.959761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.959682 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:04.975439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.975053 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "kube-api-access-zc4kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:04.985260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.977049 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.988329 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b3d355a-afae-4e36-84db-c10434e04638" containerID="436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b" exitCode=0 Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.989729 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.990230 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" event={"ID":"0b3d355a-afae-4e36-84db-c10434e04638","Type":"ContainerDied","Data":"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b"} Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.990268 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd" event={"ID":"0b3d355a-afae-4e36-84db-c10434e04638","Type":"ContainerDied","Data":"99c30e06a3bbe1b26368289d5603547403d23c8efbe0e9992e908e0b71eec197"} Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.990361 5036 scope.go:122] "RemoveContainer" containerID="436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b" Nov 01 22:16:04.994629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:04.993114 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm" pod "0b3d355a-afae-4e36-84db-c10434e04638" (UID: "0b3d355a-afae-4e36-84db-c10434e04638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:05.027289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.027264 5036 scope.go:122] "RemoveContainer" containerID="436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b" Nov 01 22:16:05.027870 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:05.027843 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b\": container with ID starting with 436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b not found: ID does not exist" containerID="436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b" Nov 01 22:16:05.027998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.027880 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b"} err="failed to get container status \"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b\": rpc error: code = NotFound desc = could not find container \"436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b\": container with ID starting with 436f99361c019284c6c460bc32dbefdcccbb8b557e364e0f9dc57ae63676ef2b not found: ID does not exist" Nov 01 22:16:05.062314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.062278 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b3d355a-afae-4e36-84db-c10434e04638-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:05.062314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.062316 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4kr\" (UniqueName: \"kubernetes.io/projected/0b3d355a-afae-4e36-84db-c10434e04638-kube-api-access-zc4kr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:05.062589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.062332 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b3d355a-afae-4e36-84db-c10434e04638-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:05.128492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.127331 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:05.254830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.254730 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:05.661465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.660628 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:16:05.679909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.672238 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-830149/test-model-830149-kserve-f44787b85-cc6wd"] Nov 01 22:16:05.903515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:05.900433 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:06.031220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:06.031171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:06.468067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:06.467955 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3d355a-afae-4e36-84db-c10434e04638" path="/var/lib/kubelet/pods/0b3d355a-afae-4e36-84db-c10434e04638/volumes" Nov 01 22:16:06.823599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:06.823563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:07.724343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:07.724310 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:08.562247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.562213 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr"] Nov 01 22:16:08.562976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.562959 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" Nov 01 22:16:08.562976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.562975 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" Nov 01 22:16:08.563099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.563089 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3d355a-afae-4e36-84db-c10434e04638" containerName="main" Nov 01 22:16:08.568514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.568491 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.572830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.572812 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:16:08.573982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.573962 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:16:08.576223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.576196 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr"] Nov 01 22:16:08.647440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.647404 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:08.656226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.656194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.656385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.656272 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.656498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.656470 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mw5\" (UniqueName: \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.760347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.760312 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mw5\" (UniqueName: \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.761482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.761434 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.761747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.761718 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.762818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.762719 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.771463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.771132 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.775945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.775921 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mw5\" (UniqueName: \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\") pod \"maas-api-revoke-sub-210b38cc-hdljr\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:08.897278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:08.897191 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:09.192537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:09.192501 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr"] Nov 01 22:16:09.192694 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:16:09.192583 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ce7343_48a7_40ec_923d_5ff711c836e7.slice/crio-6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb WatchSource:0}: Error finding container 6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb: Status 404 returned error can't find the container with id 6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb Nov 01 22:16:09.412497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:09.391054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:10.060538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:10.060499 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" event={"ID":"11ce7343-48a7-40ec-923d-5ff711c836e7","Type":"ContainerStarted","Data":"4a0564de18b67f8e3772b2c2040b7b8d8acba7a88e5d037e287dbee3de6ab17d"} Nov 01 22:16:10.061035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:10.060594 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" event={"ID":"11ce7343-48a7-40ec-923d-5ff711c836e7","Type":"ContainerStarted","Data":"6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb"} Nov 01 22:16:10.083514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:10.083460 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" podStartSLOduration=2.083428651 podStartE2EDuration="2.083428651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:10.07904634 +0000 UTC m=+1923.772026828" watchObservedRunningTime="2026-11-01 22:16:10.083428651 +0000 UTC m=+1923.776409137" Nov 01 22:16:10.131216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:10.131186 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:11.071379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:11.071349 5036 generic.go:356] "Generic (PLEG): container finished" podID="11ce7343-48a7-40ec-923d-5ff711c836e7" containerID="4a0564de18b67f8e3772b2c2040b7b8d8acba7a88e5d037e287dbee3de6ab17d" exitCode=0 Nov 01 22:16:11.071828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:11.071433 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" event={"ID":"11ce7343-48a7-40ec-923d-5ff711c836e7","Type":"ContainerDied","Data":"4a0564de18b67f8e3772b2c2040b7b8d8acba7a88e5d037e287dbee3de6ab17d"} Nov 01 22:16:11.077635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:11.077603 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:11.858266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:11.858229 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:12.277922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.277890 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:12.447189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.447113 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\") pod \"11ce7343-48a7-40ec-923d-5ff711c836e7\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " Nov 01 22:16:12.447327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.447276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\") pod \"11ce7343-48a7-40ec-923d-5ff711c836e7\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " Nov 01 22:16:12.447400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.447350 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\" (UniqueName: \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\") pod \"11ce7343-48a7-40ec-923d-5ff711c836e7\" (UID: \"11ce7343-48a7-40ec-923d-5ff711c836e7\") " Nov 01 22:16:12.447689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.447666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca" pod "11ce7343-48a7-40ec-923d-5ff711c836e7" (UID: "11ce7343-48a7-40ec-923d-5ff711c836e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:12.448041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.448022 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:12.449274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.449247 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca" pod "11ce7343-48a7-40ec-923d-5ff711c836e7" (UID: "11ce7343-48a7-40ec-923d-5ff711c836e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:12.449377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.449349 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5" pod "11ce7343-48a7-40ec-923d-5ff711c836e7" (UID: "11ce7343-48a7-40ec-923d-5ff711c836e7"). InnerVolumeSpecName "kube-api-access-t9mw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:12.556501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.552039 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mw5\" (UniqueName: \"kubernetes.io/projected/11ce7343-48a7-40ec-923d-5ff711c836e7-kube-api-access-t9mw5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:12.556501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.552086 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce7343-48a7-40ec-923d-5ff711c836e7-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:12.748522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.744837 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:12.879316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.878235 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:12.973828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:12.973775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:13.101466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:13.095900 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" Nov 01 22:16:13.101466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:13.095927 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr" event={"ID":"11ce7343-48a7-40ec-923d-5ff711c836e7","Type":"ContainerDied","Data":"6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb"} Nov 01 22:16:13.101466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:13.095988 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f83418c6054f1311ca74c1edee481d69520c06bb69841881bcfaf728b5978cb" Nov 01 22:16:13.115699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:13.115002 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:16:13.935265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:13.935221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:14.743249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:14.743211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:15.339887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:15.339844 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:15.472259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:15.472214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:16.121794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:16.121749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:16.279201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:16.279163 5036 scope.go:122] "RemoveContainer" containerID="3637b4e39eef984ad5ba3af99accd974ec95f294b9bead8b7c6dbf90b890a4b0" Nov 01 22:16:16.894236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:16.894202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:17.627480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:17.627345 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:18.347465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.347246 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:18.646357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.639619 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:16:18.646357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.640783 5036 cpu_manager.go:414] "containerMap: removing container" podUID="11ce7343-48a7-40ec-923d-5ff711c836e7" containerName="revoke-keys" Nov 01 22:16:18.646357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.640805 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ce7343-48a7-40ec-923d-5ff711c836e7" containerName="revoke-keys" Nov 01 22:16:18.646357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.641051 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ce7343-48a7-40ec-923d-5ff711c836e7" containerName="revoke-keys" Nov 01 22:16:18.652463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.650042 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.657462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.656252 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:16:18.668692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.665922 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-d6e25f\"/\"auth-test-model-d6e25f-kserve-self-signed-certs\"" Nov 01 22:16:18.668692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.667183 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-d6e25f\"/\"openshift-service-ca.crt\"" Nov 01 22:16:18.668692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.667283 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-d6e25f\"/\"kube-root-ca.crt\"" Nov 01 22:16:18.668692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.667333 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-d6e25f\"/\"default-dockercfg-ft9r5\"" Nov 01 22:16:18.694726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.694686 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.694877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.694748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tccd\" (UniqueName: \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.694877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.694793 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.695053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.695031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.695156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.695138 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.695231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.695215 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.715627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.715599 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:18.796910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.796878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.796982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.797046 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.797164 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.797194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tccd\" (UniqueName: \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.797236 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.797349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.797325 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.798071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.798044 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.798382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.798362 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.800000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.799980 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.833531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.832165 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.864468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.836789 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tccd\" (UniqueName: \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\") pod \"auth-test-model-d6e25f-kserve-576c87d997-blvbv\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:18.975139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:18.974805 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:19.196464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.182243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:19.219757 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:19.211732 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:16:19.334947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.334905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:19.378296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.378209 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq"] Nov 01 22:16:19.434557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.434335 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9d8b5d-flkvq"] Nov 01 22:16:19.447494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.446793 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:16:19.598509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.597690 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:16:19.658402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.648748 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:19.669768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.665742 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h"] Nov 01 22:16:19.675017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.674950 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17957f-srt8h"] Nov 01 22:16:19.775974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.775940 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:19.915541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:19.912726 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:20.099466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.098910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:20.276469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.274422 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:20.355468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.355034 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" event={"ID":"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d","Type":"ContainerStarted","Data":"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192"} Nov 01 22:16:20.355468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.355399 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" event={"ID":"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d","Type":"ContainerStarted","Data":"eb385f514a0b1ef025aad61aeb81404d18d3aeaa8f2b4012467d6bedc5679f45"} Nov 01 22:16:20.391465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.388663 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6"] Nov 01 22:16:20.466804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.462334 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:20.475772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.475724 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podStartSLOduration=2.475710343 podStartE2EDuration="2.475710343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:20.45327166 +0000 UTC m=+1934.146252147" watchObservedRunningTime="2026-11-01 22:16:20.475710343 +0000 UTC m=+1934.168690830" Nov 01 22:16:20.478797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.478768 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a416b131-3b53-41b4-9c3d-a1ffeb808965" path="/var/lib/kubelet/pods/a416b131-3b53-41b4-9c3d-a1ffeb808965/volumes" Nov 01 22:16:20.479484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.479463 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0436a9-938d-4fa1-9188-f1527b68b4bb" path="/var/lib/kubelet/pods/fc0436a9-938d-4fa1-9188-f1527b68b4bb/volumes" Nov 01 22:16:20.479939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.479923 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8483c2-gv5g6"] Nov 01 22:16:20.832955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.832907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:20.976386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:20.976353 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:21.428967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:21.428933 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:22.453469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:22.453275 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d592b36b-33af-4307-83d5-1c5099e905a7" path="/var/lib/kubelet/pods/d592b36b-33af-4307-83d5-1c5099e905a7/volumes" Nov 01 22:16:22.759903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:22.759798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:23.643079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:23.643042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:24.447107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:24.447070 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:25.266774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:25.266736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:25.943912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:25.943875 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:26.749080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:26.749041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:27.461762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:27.461723 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:27.765526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:27.759998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:27.893707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:27.893663 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:28.530474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:28.519798 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:29.135562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.117971 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:16:29.162466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.141142 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": context deadline exceeded" Nov 01 22:16:29.332912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.332862 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:16:29.352740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.352695 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:16:29.353054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.353018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:29.502884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.502800 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:29.539380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.539345 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:16:29.539560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.539419 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:16:29.539560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.539551 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:29.541299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.541127 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:29.542356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.542323 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" containerID="cri-o://8592cd47fb99a061529b2c7b37cbc53e5bd93e566b3603d9e272f122c7dfe98a" gracePeriod=30 Nov 01 22:16:29.543395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.542599 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" containerID="cri-o://b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1" gracePeriod=30 Nov 01 22:16:29.548431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.540803 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:16:29.552947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.552924 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:16:29.553143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.553119 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" containerID="cri-o://c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d" gracePeriod=30 Nov 01 22:16:29.561308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.561289 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl"] Nov 01 22:16:29.567819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.567796 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.571656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.571629 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl"] Nov 01 22:16:29.575804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.575781 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:16:29.575931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.575914 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:16:29.600540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.600508 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:16:29.649883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.647759 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:29.773531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.772797 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.773531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.773089 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.773531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.773162 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jq8c\" (UniqueName: \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.792469 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:29.774111 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:29.792469 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:29.774177 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:30.274157171 +0000 UTC m=+1943.967137652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:29.885989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.885938 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.886527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.886489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.886734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.886703 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jq8c\" (UniqueName: \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.912045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.911994 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:29.954536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:29.953527 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:30.033468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.030110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jq8c\" (UniqueName: \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\") pod \"maas-api-revoke-keys-e2e-inf-b-830149-fw9hl\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:30.035528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.035500 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4"] Nov 01 22:16:30.043219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.042927 5036 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:16:30.043219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.042974 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:16:30.045192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.045172 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.060456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.046519 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4"] Nov 01 22:16:30.090001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.089703 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:16:30.147508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.130510 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:30.179469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.154392 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.179809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.179775 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvv2\" (UniqueName: \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.180377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.180345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.220466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.219716 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:30.328517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.282712 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.333601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.333560 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvv2\" (UniqueName: \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.350256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.349911 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.365622 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:30.350518 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:30.365622 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:30.350602 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:31.350580813 +0000 UTC m=+1945.043561284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:30.365622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.351311 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.401260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.401224 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvv2\" (UniqueName: \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.567543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.553428 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea41e092-jkwr4\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.653920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.653878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:30.721785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.721758 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:30.938640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.935944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:30.951115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:30.951076 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl"] Nov 01 22:16:31.045927 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:16:31.045872 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b71d857_5d0e_45f3_ba93_18868b6b9371.slice/crio-b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69 WatchSource:0}: Error finding container b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69: Status 404 returned error can't find the container with id b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69 Nov 01 22:16:31.048885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.048860 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4"] Nov 01 22:16:31.411734 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:31.411658 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:31.411734 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:31.411732 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:33.411718486 +0000 UTC m=+1947.104698949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:31.553089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.553043 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" event={"ID":"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6","Type":"ContainerStarted","Data":"e32e5c3a26d67e18b514fc7c2d7194d70cacdf50887113ab955aea3043ac9ae6"} Nov 01 22:16:31.553252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.553107 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" event={"ID":"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6","Type":"ContainerStarted","Data":"8d25f3689379461b2e14b03f92a34cc611c46bf116b12781fca9e94ffe3181b8"} Nov 01 22:16:31.555143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.555086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" event={"ID":"0b71d857-5d0e-45f3-ba93-18868b6b9371","Type":"ContainerStarted","Data":"9a1f40a36992511b25c3958d608e6cca07b67f57569b5a299b8dfaef02631570"} Nov 01 22:16:31.555294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.555168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" event={"ID":"0b71d857-5d0e-45f3-ba93-18868b6b9371","Type":"ContainerStarted","Data":"b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69"} Nov 01 22:16:31.577169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.577109 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" podStartSLOduration=2.577089543 podStartE2EDuration="2.577089543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:31.573782954 +0000 UTC m=+1945.266763440" watchObservedRunningTime="2026-11-01 22:16:31.577089543 +0000 UTC m=+1945.270070031" Nov 01 22:16:31.595430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.595375 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" podStartSLOduration=2.595362173 podStartE2EDuration="2.595362173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:31.594533251 +0000 UTC m=+1945.287513738" watchObservedRunningTime="2026-11-01 22:16:31.595362173 +0000 UTC m=+1945.288342708" Nov 01 22:16:31.626400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:31.626365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:32.433916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:32.433879 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:32.566937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:32.566899 5036 generic.go:356] "Generic (PLEG): container finished" podID="0b71d857-5d0e-45f3-ba93-18868b6b9371" containerID="9a1f40a36992511b25c3958d608e6cca07b67f57569b5a299b8dfaef02631570" exitCode=0 Nov 01 22:16:32.567221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:32.567197 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" event={"ID":"0b71d857-5d0e-45f3-ba93-18868b6b9371","Type":"ContainerDied","Data":"9a1f40a36992511b25c3958d608e6cca07b67f57569b5a299b8dfaef02631570"} Nov 01 22:16:32.570165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:32.569798 5036 generic.go:356] "Generic (PLEG): container finished" podID="d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" containerID="e32e5c3a26d67e18b514fc7c2d7194d70cacdf50887113ab955aea3043ac9ae6" exitCode=0 Nov 01 22:16:32.570165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:32.569894 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" event={"ID":"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6","Type":"ContainerDied","Data":"e32e5c3a26d67e18b514fc7c2d7194d70cacdf50887113ab955aea3043ac9ae6"} Nov 01 22:16:33.249650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.249605 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:33.460834 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:33.460806 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:33.461230 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:33.460874 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:37.460859202 +0000 UTC m=+1951.153839666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:33.833998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.833914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:33.834578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.834561 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:33.898018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.897980 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:33.972136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972094 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\" (UniqueName: \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\") pod \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " Nov 01 22:16:33.972336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972278 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\") pod \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " Nov 01 22:16:33.972516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972493 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\") pod \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\" (UID: \"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6\") " Nov 01 22:16:33.972676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972657 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\") pod \"0b71d857-5d0e-45f3-ba93-18868b6b9371\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " Nov 01 22:16:33.972749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972706 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\" (UniqueName: \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\") pod \"0b71d857-5d0e-45f3-ba93-18868b6b9371\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " Nov 01 22:16:33.972839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972793 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\") pod \"0b71d857-5d0e-45f3-ba93-18868b6b9371\" (UID: \"0b71d857-5d0e-45f3-ba93-18868b6b9371\") " Nov 01 22:16:33.972963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.972932 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca" pod "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" (UID: "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:33.973133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.973109 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca" pod "0b71d857-5d0e-45f3-ba93-18868b6b9371" (UID: "0b71d857-5d0e-45f3-ba93-18868b6b9371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:33.973819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.973790 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.973938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.973823 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.974999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.974971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c" pod "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" (UID: "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6"). InnerVolumeSpecName "kube-api-access-2jq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:33.976215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.976186 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:33.990865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.988591 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca" pod "0b71d857-5d0e-45f3-ba93-18868b6b9371" (UID: "0b71d857-5d0e-45f3-ba93-18868b6b9371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:33.990865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.989803 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca" pod "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" (UID: "d87d0383-8bf4-4a74-9103-24c2ad8ee4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:33.990865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:33.989907 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2" pod "0b71d857-5d0e-45f3-ba93-18868b6b9371" (UID: "0b71d857-5d0e-45f3-ba93-18868b6b9371"). InnerVolumeSpecName "kube-api-access-5vvv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:34.075564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.075516 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvv2\" (UniqueName: \"kubernetes.io/projected/0b71d857-5d0e-45f3-ba93-18868b6b9371-kube-api-access-5vvv2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.075564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.075562 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b71d857-5d0e-45f3-ba93-18868b6b9371-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.075785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.075580 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jq8c\" (UniqueName: \"kubernetes.io/projected/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-kube-api-access-2jq8c\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.075785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.075596 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.137666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.137578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:34.598013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.597977 5036 generic.go:356] "Generic (PLEG): container finished" podID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerID="8592cd47fb99a061529b2c7b37cbc53e5bd93e566b3603d9e272f122c7dfe98a" exitCode=0 Nov 01 22:16:34.598535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.598039 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" event={"ID":"50cd8a73-70f3-448a-9dc8-2598d4a2c52f","Type":"ContainerDied","Data":"8592cd47fb99a061529b2c7b37cbc53e5bd93e566b3603d9e272f122c7dfe98a"} Nov 01 22:16:34.601203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.601165 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" Nov 01 22:16:34.611610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.602952 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl" event={"ID":"d87d0383-8bf4-4a74-9103-24c2ad8ee4b6","Type":"ContainerDied","Data":"8d25f3689379461b2e14b03f92a34cc611c46bf116b12781fca9e94ffe3181b8"} Nov 01 22:16:34.611610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.603042 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d25f3689379461b2e14b03f92a34cc611c46bf116b12781fca9e94ffe3181b8" Nov 01 22:16:34.616681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.616066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" event={"ID":"0b71d857-5d0e-45f3-ba93-18868b6b9371","Type":"ContainerDied","Data":"b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69"} Nov 01 22:16:34.616681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.616119 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0af13a378a7e16ce73e513609325b6ea03d0387c0faf5524bc8f27c8f788f69" Nov 01 22:16:34.616681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.616227 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4" Nov 01 22:16:34.938696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.938670 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:16:34.999277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:34.999248 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:16:35.037027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.036987 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037041 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037080 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037157 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037189 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037223 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037317 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037369 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037433 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037607 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037733 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.037873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.037873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037864 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\") pod \"a40b676b-5ad7-423d-9223-abed4d646fee\" (UID: \"a40b676b-5ad7-423d-9223-abed4d646fee\") " Nov 01 22:16:35.038030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.037921 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.038030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.038008 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.038183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.038163 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\") pod \"941318d2-f54e-4934-9c9c-7267bcc75f54\" (UID: \"941318d2-f54e-4934-9c9c-7267bcc75f54\") " Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039180 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039313 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039786 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039960 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.039991 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.040149 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.040314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.040801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.040634 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:35.049541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.044838 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:35.049541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.045637 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:35.049541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.046987 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.049541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.047534 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "kube-api-access-2cxsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.050931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.050892 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.050931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.050894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.051156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.051134 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47" pod "941318d2-f54e-4934-9c9c-7267bcc75f54" (UID: "941318d2-f54e-4934-9c9c-7267bcc75f54"). InnerVolumeSpecName "kube-api-access-75c47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.052997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.052965 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token" pod "a40b676b-5ad7-423d-9223-abed4d646fee" (UID: "a40b676b-5ad7-423d-9223-abed4d646fee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.122869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.122832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:35.140974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.140936 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941318d2-f54e-4934-9c9c-7267bcc75f54-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.140974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.140975 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75c47\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-kube-api-access-75c47\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.140993 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b676b-5ad7-423d-9223-abed4d646fee-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141007 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141020 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141035 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxsf\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-kube-api-access-2cxsf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141048 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141063 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141076 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b676b-5ad7-423d-9223-abed4d646fee-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141091 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141105 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141118 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141131 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141147 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141159 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b676b-5ad7-423d-9223-abed4d646fee-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141173 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b676b-5ad7-423d-9223-abed4d646fee-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141186 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.141204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.141199 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941318d2-f54e-4934-9c9c-7267bcc75f54-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.376469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.375759 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:35.467469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.466827 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:16:35.636682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.636593 5036 generic.go:356] "Generic (PLEG): container finished" podID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerID="b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1" exitCode=0 Nov 01 22:16:35.636682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.636655 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" Nov 01 22:16:35.637235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.636658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" event={"ID":"941318d2-f54e-4934-9c9c-7267bcc75f54","Type":"ContainerDied","Data":"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1"} Nov 01 22:16:35.637235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.636867 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz" event={"ID":"941318d2-f54e-4934-9c9c-7267bcc75f54","Type":"ContainerDied","Data":"ef1b0451d54df3276cc0e1f0ac685e5c89c3a7f52dd6982a1bf5469c2f86279b"} Nov 01 22:16:35.637235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.636903 5036 scope.go:122] "RemoveContainer" containerID="b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1" Nov 01 22:16:35.639627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.639600 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.640160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.640067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.640293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.640145 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.640547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.640526 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.640732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.640713 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.641027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641003 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.641182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641163 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.641557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641535 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.641706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641674 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" Nov 01 22:16:35.644625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerDied","Data":"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d"} Nov 01 22:16:35.644737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.644737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.644686 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\") pod \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\" (UID: \"50cd8a73-70f3-448a-9dc8-2598d4a2c52f\") " Nov 01 22:16:35.644919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.640728 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:35.644919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641916 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.644919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.644857 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:35.645208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.645521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645496 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.645621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645526 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.645621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645541 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.645621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645555 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.645621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645574 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.645621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.645932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.645784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.645932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.641569 5036 generic.go:356] "Generic (PLEG): container finished" podID="a40b676b-5ad7-423d-9223-abed4d646fee" containerID="c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d" exitCode=0 Nov 01 22:16:35.646076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.646056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm" event={"ID":"a40b676b-5ad7-423d-9223-abed4d646fee","Type":"ContainerDied","Data":"30479df2115124d2aa9e7eb68913c0a3494e6319e15d7e67190253a32d95d83d"} Nov 01 22:16:35.648251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.648224 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:35.648554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.648530 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" event={"ID":"50cd8a73-70f3-448a-9dc8-2598d4a2c52f","Type":"ContainerDied","Data":"55c932b4cb32e771356dcdbaac26c93461e4175be891386c76bdbeb80b5ffe2a"} Nov 01 22:16:35.648628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.648564 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr" Nov 01 22:16:35.651669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.651632 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw" pod "50cd8a73-70f3-448a-9dc8-2598d4a2c52f" (UID: "50cd8a73-70f3-448a-9dc8-2598d4a2c52f"). InnerVolumeSpecName "kube-api-access-csfvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:35.653806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.653747 5036 scope.go:122] "RemoveContainer" containerID="b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1" Nov 01 22:16:35.654125 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:35.654097 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1\": container with ID starting with b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1 not found: ID does not exist" containerID="b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1" Nov 01 22:16:35.654233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.654133 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1"} err="failed to get container status \"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1\": rpc error: code = NotFound desc = could not find container \"b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1\": container with ID starting with b12bcb227fe0451884f1a82bdb327ec4e0a0c198e149b5a9f4007d6b4a969cc1 not found: ID does not exist" Nov 01 22:16:35.654233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.654194 5036 scope.go:122] "RemoveContainer" containerID="c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d" Nov 01 22:16:35.666068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.666053 5036 scope.go:122] "RemoveContainer" containerID="1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368" Nov 01 22:16:35.676351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.676165 5036 scope.go:122] "RemoveContainer" containerID="c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d" Nov 01 22:16:35.676531 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:35.676435 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d\": container with ID starting with c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d not found: ID does not exist" containerID="c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d" Nov 01 22:16:35.676628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.676541 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d"} err="failed to get container status \"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d\": rpc error: code = NotFound desc = could not find container \"c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d\": container with ID starting with c438551de42d6e7bd532dccfd68cbb89b292a833ec664a1b8068f46011fb9b6d not found: ID does not exist" Nov 01 22:16:35.676628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.676562 5036 scope.go:122] "RemoveContainer" containerID="1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368" Nov 01 22:16:35.676872 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:35.676854 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368\": container with ID starting with 1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368 not found: ID does not exist" containerID="1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368" Nov 01 22:16:35.676935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.676876 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368"} err="failed to get container status \"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368\": rpc error: code = NotFound desc = could not find container \"1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368\": container with ID starting with 1ed4602b1b44db66ff51e6a0cd7628d56f3a26dfa7c7e7688b35ddd89b16b368 not found: ID does not exist" Nov 01 22:16:35.676935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.676889 5036 scope.go:122] "RemoveContainer" containerID="8592cd47fb99a061529b2c7b37cbc53e5bd93e566b3603d9e272f122c7dfe98a" Nov 01 22:16:35.689053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.688984 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:16:35.695430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.695407 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-zhgcm"] Nov 01 22:16:35.704957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.704925 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:16:35.709543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.709522 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-5qbdz"] Nov 01 22:16:35.747453 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.747426 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.747562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.747464 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.747562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.747474 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.747562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.747484 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfvw\" (UniqueName: \"kubernetes.io/projected/50cd8a73-70f3-448a-9dc8-2598d4a2c52f-kube-api-access-csfvw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.977779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.977744 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:16:35.989517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:35.989488 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-tg8sr"] Nov 01 22:16:36.149346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:36.149313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:36.441291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:36.441253 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" path="/var/lib/kubelet/pods/50cd8a73-70f3-448a-9dc8-2598d4a2c52f/volumes" Nov 01 22:16:36.441866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:36.441845 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" path="/var/lib/kubelet/pods/941318d2-f54e-4934-9c9c-7267bcc75f54/volumes" Nov 01 22:16:36.442323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:36.442309 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" path="/var/lib/kubelet/pods/a40b676b-5ad7-423d-9223-abed4d646fee/volumes" Nov 01 22:16:36.838049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:36.838013 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:37.483171 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:37.482975 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:37.483171 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:37.483065 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:45.483046664 +0000 UTC m=+1959.176027141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:37.630627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:37.630573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:38.470811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.470742 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:38.502468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.502168 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503513 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503539 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503560 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0b71d857-5d0e-45f3-ba93-18868b6b9371" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503602 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b71d857-5d0e-45f3-ba93-18868b6b9371" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503622 5036 cpu_manager.go:414] "containerMap: removing container" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503629 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503658 5036 cpu_manager.go:414] "containerMap: removing container" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503666 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503686 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503693 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503911 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503923 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b71d857-5d0e-45f3-ba93-18868b6b9371" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503934 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cd8a73-70f3-448a-9dc8-2598d4a2c52f" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503948 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="941318d2-f54e-4934-9c9c-7267bcc75f54" containerName="istio-proxy" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503960 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" containerName="revoke-keys" Nov 01 22:16:38.506465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.503975 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.515946 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.516058 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.526078 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-702002\"/\"openshift-service-ca.crt\"" Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.526153 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-702002\"/\"kube-root-ca.crt\"" Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.526231 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-702002\"/\"default-dockercfg-rt9gt\"" Nov 01 22:16:38.528495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.526255 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-702002\"/\"auth-test-model-702002-kserve-self-signed-certs\"" Nov 01 22:16:38.602278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.602236 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:38.608336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608307 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.608548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608437 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfg6\" (UniqueName: \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.608548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608505 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.608786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608761 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.608897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608822 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.608897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.608850 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.728516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.711405 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.728816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.728786 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.728969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.728954 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.729315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.729293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.729524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.729504 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfg6\" (UniqueName: \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.744465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.729692 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.745319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.745293 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.759650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.745759 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.826653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.754367 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.826932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.755335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.827046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.771261 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.827142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.794310 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfg6\" (UniqueName: \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\") pod \"auth-test-model-702002-kserve-5f969cdd5c-9sdhb\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:38.841289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:38.840948 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:39.250364 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:16:39.250330 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1179ed2c_c163_4c42_827b_2838746f0f43.slice/crio-82ee40b1c1b4db45611bac27e1fd032d3a69c2eb2d8f67cd373c8e67d25c111b WatchSource:0}: Error finding container 82ee40b1c1b4db45611bac27e1fd032d3a69c2eb2d8f67cd373c8e67d25c111b: Status 404 returned error can't find the container with id 82ee40b1c1b4db45611bac27e1fd032d3a69c2eb2d8f67cd373c8e67d25c111b Nov 01 22:16:39.251473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.251413 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:16:39.254505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.254476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:39.547056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.547008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:39.685892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.685845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:39.873523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.872209 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" event={"ID":"1179ed2c-c163-4c42-827b-2838746f0f43","Type":"ContainerStarted","Data":"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b"} Nov 01 22:16:39.873523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.872281 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" event={"ID":"1179ed2c-c163-4c42-827b-2838746f0f43","Type":"ContainerStarted","Data":"82ee40b1c1b4db45611bac27e1fd032d3a69c2eb2d8f67cd373c8e67d25c111b"} Nov 01 22:16:39.939469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:39.938149 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podStartSLOduration=1.938127022 podStartE2EDuration="1.938127022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:39.903932384 +0000 UTC m=+1953.596912871" watchObservedRunningTime="2026-11-01 22:16:39.938127022 +0000 UTC m=+1953.631107510" Nov 01 22:16:40.414905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:40.414863 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:40.842141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:40.842112 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:41.058957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:41.058917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:41.740236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:41.740195 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:42.444503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:42.444423 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:43.247728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:43.247674 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:44.046704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:44.046655 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:44.713088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:44.713029 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:44.900656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:44.900614 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:44.930983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:44.930936 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl"] Nov 01 22:16:44.994516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:44.968772 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-830149-fw9hl"] Nov 01 22:16:45.065472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.056952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:45.377056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.376830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:45.527628 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:45.527596 5036 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-self-signed-certs: secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:45.527801 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:45.527673 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs podName:6c98d127-a563-4526-ba06-1c87eac0fb85 nodeName:}" failed. No retries permitted until 2026-11-01 22:17:01.527654166 +0000 UTC m=+1975.220634642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs") pod "test-model-487639-kserve-5fbcd86d85-xms2k" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85") : secret "test-model-487639-kserve-self-signed-certs" not found Nov 01 22:16:45.597739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.597690 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" containerID="cri-o://866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273" gracePeriod=45 Nov 01 22:16:45.628028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.627995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": dial tcp 10.132.0.231:8000: connect: connection refused" Nov 01 22:16:45.926295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.926264 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:16:45.993256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.993230 5036 generic.go:356] "Generic (PLEG): container finished" podID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerID="866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273" exitCode=0 Nov 01 22:16:45.993583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.993329 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" Nov 01 22:16:45.993728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.993348 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" event={"ID":"6c98d127-a563-4526-ba06-1c87eac0fb85","Type":"ContainerDied","Data":"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273"} Nov 01 22:16:45.993838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.993818 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k" event={"ID":"6c98d127-a563-4526-ba06-1c87eac0fb85","Type":"ContainerDied","Data":"8a02ec640c1664b70c8aa19c6b854a5ff365bc89dee5443ed101c8df5b6c71ac"} Nov 01 22:16:45.993937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:45.993850 5036 scope.go:122] "RemoveContainer" containerID="866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273" Nov 01 22:16:46.005754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.005723 5036 scope.go:122] "RemoveContainer" containerID="866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273" Nov 01 22:16:46.006067 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:46.006034 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273\": container with ID starting with 866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273 not found: ID does not exist" containerID="866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273" Nov 01 22:16:46.006158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.006078 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273"} err="failed to get container status \"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273\": rpc error: code = NotFound desc = could not find container \"866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273\": container with ID starting with 866532226a7ae129007d54db62a9168d0d6c2050c73b6cccc8aa7e2856f2c273 not found: ID does not exist" Nov 01 22:16:46.022092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.022020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:46.058677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.052966 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.058677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.053152 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.058677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.053204 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.058677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.053363 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\" (UniqueName: \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.072637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.072593 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.073187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.072689 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\") pod \"6c98d127-a563-4526-ba06-1c87eac0fb85\" (UID: \"6c98d127-a563-4526-ba06-1c87eac0fb85\") " Nov 01 22:16:46.077044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.073885 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:46.084894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.084480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:46.106594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.103751 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:46.107064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.107021 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:46.129116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.127901 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:46.129116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.127951 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx" pod "6c98d127-a563-4526-ba06-1c87eac0fb85" (UID: "6c98d127-a563-4526-ba06-1c87eac0fb85"). InnerVolumeSpecName "kube-api-access-42pdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:46.175037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.174968 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.175037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.175000 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c98d127-a563-4526-ba06-1c87eac0fb85-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.175037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.175014 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.175037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.175026 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.175258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.175040 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pdx\" (UniqueName: \"kubernetes.io/projected/6c98d127-a563-4526-ba06-1c87eac0fb85-kube-api-access-42pdx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.175258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.175054 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c98d127-a563-4526-ba06-1c87eac0fb85-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:46.327310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.327277 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:16:46.331907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.331878 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-487639/test-model-487639-kserve-5fbcd86d85-xms2k"] Nov 01 22:16:46.441037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.440970 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" path="/var/lib/kubelet/pods/6c98d127-a563-4526-ba06-1c87eac0fb85/volumes" Nov 01 22:16:46.441438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.441423 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87d0383-8bf4-4a74-9103-24c2ad8ee4b6" path="/var/lib/kubelet/pods/d87d0383-8bf4-4a74-9103-24c2ad8ee4b6/volumes" Nov 01 22:16:46.666307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:46.666265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:47.369571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:47.369523 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:48.250999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:48.250961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:48.842464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:48.842027 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:48.861873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:48.861835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:48.863555 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:16:48.863527 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:16:49.014481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.014401 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:49.090241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.090206 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:16:49.492066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.492018 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:16:49.503522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.494700 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:49.503522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.494727 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b676b-5ad7-423d-9223-abed4d646fee" containerName="istio-proxy" Nov 01 22:16:49.503522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.494869 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" Nov 01 22:16:49.503522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.494882 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" Nov 01 22:16:49.503522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.495156 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c98d127-a563-4526-ba06-1c87eac0fb85" containerName="main" Nov 01 22:16:49.506251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.506207 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:16:49.506565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.506529 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.515306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.515282 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-da34ca\"/\"auth-test-model-da34ca-kserve-self-signed-certs\"" Nov 01 22:16:49.519539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.516960 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-da34ca\"/\"openshift-service-ca.crt\"" Nov 01 22:16:49.519689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.517036 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-da34ca\"/\"kube-root-ca.crt\"" Nov 01 22:16:49.519983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.519966 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-da34ca\"/\"default-dockercfg-cvrzv\"" Nov 01 22:16:49.539522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539352 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.539522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.539522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539506 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.539781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.539846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539807 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7zt\" (UniqueName: \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.539907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.539876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.571756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.566711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:49.641079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641047 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.641079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641083 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.641355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641230 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.641427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641374 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7zt\" (UniqueName: \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.641427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641412 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.641565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.642000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.641966 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.642154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.642019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.642304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.642283 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.644740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.644707 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.645391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.645369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.655729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.655700 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7zt\" (UniqueName: \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\") pod \"auth-test-model-da34ca-kserve-578f7b449c-4gprb\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.853471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.832415 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:49.913433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:49.911616 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:50.268461 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:16:50.268420 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fae7803_f768_4c60_8328_abe5665107e0.slice/crio-d541124a42526179cf8094e314d23bb381b8b236430a3c0493f70ecf84c885cb WatchSource:0}: Error finding container d541124a42526179cf8094e314d23bb381b8b236430a3c0493f70ecf84c885cb: Status 404 returned error can't find the container with id d541124a42526179cf8094e314d23bb381b8b236430a3c0493f70ecf84c885cb Nov 01 22:16:50.268618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:50.268593 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:16:50.639307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:50.639272 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:51.057228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.057187 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" event={"ID":"2fae7803-f768-4c60-8328-abe5665107e0","Type":"ContainerStarted","Data":"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d"} Nov 01 22:16:51.057739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.057242 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" event={"ID":"2fae7803-f768-4c60-8328-abe5665107e0","Type":"ContainerStarted","Data":"d541124a42526179cf8094e314d23bb381b8b236430a3c0493f70ecf84c885cb"} Nov 01 22:16:51.081254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.081203 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podStartSLOduration=2.081189318 podStartE2EDuration="2.081189318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:51.080489615 +0000 UTC m=+1964.773470101" watchObservedRunningTime="2026-11-01 22:16:51.081189318 +0000 UTC m=+1964.774169849" Nov 01 22:16:51.730255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.730214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:51.833819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.833781 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:51.864179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:51.860957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:52.069201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.069163 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:52.202502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.202464 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:52.480520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.454382 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:52.848646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.848423 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:16:52.852858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.852836 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:16:52.853049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.853038 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:16:52.859742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.859708 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" containerID="cri-o://6c0b36972c4c85dab0bb1aeaf2ddf66f90020229c096968a79d624c4a8bc845e" gracePeriod=30 Nov 01 22:16:52.888946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.888886 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" containerID="cri-o://3d9e0b4c16a46e3973a2408630d9c1a70d91987c8a8930eef5f45ba86210c7e7" gracePeriod=30 Nov 01 22:16:52.931439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.922101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:52.955646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:52.955603 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" containerID="cri-o://263480c0f00375b79cc7941c9e23c4ec57c8f72ad08b00877a26c18e3e2afc3f" gracePeriod=30 Nov 01 22:16:53.664827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:53.661209 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:54.429484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:54.429401 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:55.100780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.100737 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:16:55.101243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.100805 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:16:55.248052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.247988 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:55.693116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.693063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:55.798032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.797999 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:16:55.857150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:55.857101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:56.098997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.098958 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r"] Nov 01 22:16:56.108502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.108423 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.109993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.109846 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r"] Nov 01 22:16:56.115165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.115139 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:16:56.117749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.117727 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:16:56.184505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.184415 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.184836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.184809 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.184974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.184945 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jm8\" (UniqueName: \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.276772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.276624 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:56.287066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.287025 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.287336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.287116 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jm8\" (UniqueName: \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.287702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.287596 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.288252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.288152 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.291160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.291130 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.302685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.301603 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jm8\" (UniqueName: \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\") pod \"maas-api-revoke-keys-e2e-inf-a-487639-mrl5r\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.432213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.432125 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:56.727769 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:16:56.727733 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036d384d_5abc_4b20_a40c_090e88539d21.slice/crio-c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332 WatchSource:0}: Error finding container c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332: Status 404 returned error can't find the container with id c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332 Nov 01 22:16:56.730688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.730662 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r"] Nov 01 22:16:56.841123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.841089 5036 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:16:56.841281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:56.841151 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:16:57.125115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:57.125076 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" event={"ID":"036d384d-5abc-4b20-a40c-090e88539d21","Type":"ContainerStarted","Data":"706382ae5fa4bc6fc8126c39d4dad9200170569762094b6b09d6c05cc38145ff"} Nov 01 22:16:57.125525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:57.125141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" event={"ID":"036d384d-5abc-4b20-a40c-090e88539d21","Type":"ContainerStarted","Data":"c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332"} Nov 01 22:16:57.149192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:57.149151 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" podStartSLOduration=1.14913926 podStartE2EDuration="1.14913926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:57.143049858 +0000 UTC m=+1970.836030336" watchObservedRunningTime="2026-11-01 22:16:57.14913926 +0000 UTC m=+1970.842119745" Nov 01 22:16:57.199190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:57.199155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:57.862914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:57.862823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:58.173056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.168335 5036 generic.go:356] "Generic (PLEG): container finished" podID="5628ebf8-09d6-4d65-8346-093194994d8b" containerID="3d9e0b4c16a46e3973a2408630d9c1a70d91987c8a8930eef5f45ba86210c7e7" exitCode=0 Nov 01 22:16:58.173056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.168728 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" event={"ID":"5628ebf8-09d6-4d65-8346-093194994d8b","Type":"ContainerDied","Data":"3d9e0b4c16a46e3973a2408630d9c1a70d91987c8a8930eef5f45ba86210c7e7"} Nov 01 22:16:58.173056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.173012 5036 generic.go:356] "Generic (PLEG): container finished" podID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerID="263480c0f00375b79cc7941c9e23c4ec57c8f72ad08b00877a26c18e3e2afc3f" exitCode=0 Nov 01 22:16:58.174289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.173097 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" event={"ID":"f8b6afa6-2296-4256-9802-3bdc72f29da7","Type":"ContainerDied","Data":"263480c0f00375b79cc7941c9e23c4ec57c8f72ad08b00877a26c18e3e2afc3f"} Nov 01 22:16:58.176182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.176043 5036 generic.go:356] "Generic (PLEG): container finished" podID="5d581ae5-b177-4a34-b112-f612ba985df8" containerID="6c0b36972c4c85dab0bb1aeaf2ddf66f90020229c096968a79d624c4a8bc845e" exitCode=0 Nov 01 22:16:58.176182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.176127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" event={"ID":"5d581ae5-b177-4a34-b112-f612ba985df8","Type":"ContainerDied","Data":"6c0b36972c4c85dab0bb1aeaf2ddf66f90020229c096968a79d624c4a8bc845e"} Nov 01 22:16:58.179346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.179137 5036 generic.go:356] "Generic (PLEG): container finished" podID="036d384d-5abc-4b20-a40c-090e88539d21" containerID="706382ae5fa4bc6fc8126c39d4dad9200170569762094b6b09d6c05cc38145ff" exitCode=0 Nov 01 22:16:58.179346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.179195 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" event={"ID":"036d384d-5abc-4b20-a40c-090e88539d21","Type":"ContainerDied","Data":"706382ae5fa4bc6fc8126c39d4dad9200170569762094b6b09d6c05cc38145ff"} Nov 01 22:16:58.263508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.256830 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:16:58.355012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.354981 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355174 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355382 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355465 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355512 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355551 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355627 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355663 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.355759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.355747 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\") pod \"5d581ae5-b177-4a34-b112-f612ba985df8\" (UID: \"5d581ae5-b177-4a34-b112-f612ba985df8\") " Nov 01 22:16:58.356273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.356248 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.356585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.356563 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.356661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.356588 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:58.356786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.356768 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.356864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.356831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.359912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.359770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.360826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.360742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.362501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.362479 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "kube-api-access-lv9mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.364159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.364132 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:58.370758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.370666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token" pod "5d581ae5-b177-4a34-b112-f612ba985df8" (UID: "5d581ae5-b177-4a34-b112-f612ba985df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.420960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.420649 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:16:58.424940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.424915 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:16:58.457335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457309 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457469 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457506 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.457719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457698 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457748 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457795 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.457887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.457942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457914 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.458007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457950 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.458007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.457982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\") pod \"5628ebf8-09d6-4d65-8346-093194994d8b\" (UID: \"5628ebf8-09d6-4d65-8346-093194994d8b\") " Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458682 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458705 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458715 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d581ae5-b177-4a34-b112-f612ba985df8-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458731 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d581ae5-b177-4a34-b112-f612ba985df8-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458742 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458752 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9mp\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-kube-api-access-lv9mp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458761 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d581ae5-b177-4a34-b112-f612ba985df8-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458770 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458778 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.458785 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d581ae5-b177-4a34-b112-f612ba985df8-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.459491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.459438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.459886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.459866 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:58.460204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.460180 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.460376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.460351 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.461007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.460964 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.463761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.463723 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:58.474072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.474036 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt" pod "5628ebf8-09d6-4d65-8346-093194994d8b" (UID: "5628ebf8-09d6-4d65-8346-093194994d8b"). InnerVolumeSpecName "kube-api-access-44tzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.559870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.559816 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.559922 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.559982 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560063 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560064 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.560248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560096 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560179 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560330 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560469 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560522 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\") pod \"f8b6afa6-2296-4256-9802-3bdc72f29da7\" (UID: \"f8b6afa6-2296-4256-9802-3bdc72f29da7\") " Nov 01 22:16:58.560646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.560602 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:58.562032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.561128 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.562032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.561247 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.562032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.561676 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.562032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.561812 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562032 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b6afa6-2296-4256-9802-3bdc72f29da7-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562057 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562071 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562086 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5628ebf8-09d6-4d65-8346-093194994d8b-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562101 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562114 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562128 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562142 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5628ebf8-09d6-4d65-8346-093194994d8b-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562157 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562170 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562184 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44tzt\" (UniqueName: \"kubernetes.io/projected/5628ebf8-09d6-4d65-8346-093194994d8b-kube-api-access-44tzt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.562370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.562200 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5628ebf8-09d6-4d65-8346-093194994d8b-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.563356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.563333 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "kube-api-access-fx9gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.563704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.563678 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.564379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.564337 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:58.564960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.564936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token" pod "f8b6afa6-2296-4256-9802-3bdc72f29da7" (UID: "f8b6afa6-2296-4256-9802-3bdc72f29da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.663870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.663782 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.663870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.663816 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.663870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.663829 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.663870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:58.663842 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx9gg\" (UniqueName: \"kubernetes.io/projected/f8b6afa6-2296-4256-9802-3bdc72f29da7-kube-api-access-fx9gg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:59.190897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.190867 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" Nov 01 22:16:59.191424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.190866 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx" event={"ID":"f8b6afa6-2296-4256-9802-3bdc72f29da7","Type":"ContainerDied","Data":"b6531d1f2ab0e26e546a53a41b508d43e804261530f2b403172dab9158cb7908"} Nov 01 22:16:59.191424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.191012 5036 scope.go:122] "RemoveContainer" containerID="263480c0f00375b79cc7941c9e23c4ec57c8f72ad08b00877a26c18e3e2afc3f" Nov 01 22:16:59.192976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.192955 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" Nov 01 22:16:59.193146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.192969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p" event={"ID":"5d581ae5-b177-4a34-b112-f612ba985df8","Type":"ContainerDied","Data":"964141e254aa37199e40c3b2f5cee6cd7d57c48ee4c899be4c3ca293cd522c53"} Nov 01 22:16:59.200466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.195517 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" event={"ID":"5628ebf8-09d6-4d65-8346-093194994d8b","Type":"ContainerDied","Data":"6d303dbee89c76171479b7a57cd1281bf7534e733e941102eeba4d0e3998b7b2"} Nov 01 22:16:59.200466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.195755 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt" Nov 01 22:16:59.206977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.206882 5036 scope.go:122] "RemoveContainer" containerID="6c0b36972c4c85dab0bb1aeaf2ddf66f90020229c096968a79d624c4a8bc845e" Nov 01 22:16:59.221291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.221268 5036 scope.go:122] "RemoveContainer" containerID="3d9e0b4c16a46e3973a2408630d9c1a70d91987c8a8930eef5f45ba86210c7e7" Nov 01 22:16:59.221759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.221739 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:16:59.227258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.227233 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-dvm2p"] Nov 01 22:16:59.246331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.246290 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:16:59.246902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.246873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:59.265650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.257501 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-t8bqt"] Nov 01 22:16:59.268540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.268502 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:16:59.295138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.293321 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-972dx"] Nov 01 22:16:59.470155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.470127 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:16:59.580614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.580577 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\") pod \"036d384d-5abc-4b20-a40c-090e88539d21\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " Nov 01 22:16:59.580785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.580683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\") pod \"036d384d-5abc-4b20-a40c-090e88539d21\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " Nov 01 22:16:59.580903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.580876 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\" (UniqueName: \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\") pod \"036d384d-5abc-4b20-a40c-090e88539d21\" (UID: \"036d384d-5abc-4b20-a40c-090e88539d21\") " Nov 01 22:16:59.581070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.580998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca" pod "036d384d-5abc-4b20-a40c-090e88539d21" (UID: "036d384d-5abc-4b20-a40c-090e88539d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:59.581515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.581495 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:59.583006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.582980 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca" pod "036d384d-5abc-4b20-a40c-090e88539d21" (UID: "036d384d-5abc-4b20-a40c-090e88539d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:59.583206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.583186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8" pod "036d384d-5abc-4b20-a40c-090e88539d21" (UID: "036d384d-5abc-4b20-a40c-090e88539d21"). InnerVolumeSpecName "kube-api-access-26jm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:59.683416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.683383 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036d384d-5abc-4b20-a40c-090e88539d21-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:59.683416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.683412 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26jm8\" (UniqueName: \"kubernetes.io/projected/036d384d-5abc-4b20-a40c-090e88539d21-kube-api-access-26jm8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:59.835108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.834514 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:59.856286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.856260 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:16:59.874363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.874336 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:16:59.962250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.962184 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:16:59.963612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.963582 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" Nov 01 22:16:59.963814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.963783 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" Nov 01 22:16:59.963966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.963951 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" Nov 01 22:16:59.964056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964042 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" Nov 01 22:16:59.964169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964157 5036 cpu_manager.go:414] "containerMap: removing container" podUID="036d384d-5abc-4b20-a40c-090e88539d21" containerName="revoke-keys" Nov 01 22:16:59.964252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964239 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036d384d-5abc-4b20-a40c-090e88539d21" containerName="revoke-keys" Nov 01 22:16:59.964352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964342 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" Nov 01 22:16:59.964430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964418 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" Nov 01 22:16:59.964740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964720 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" containerName="istio-proxy" Nov 01 22:16:59.964880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964854 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" containerName="istio-proxy" Nov 01 22:16:59.964982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.964969 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="036d384d-5abc-4b20-a40c-090e88539d21" containerName="revoke-keys" Nov 01 22:16:59.965080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.965069 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" containerName="istio-proxy" Nov 01 22:16:59.975765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.975741 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:16:59.977513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.977489 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:16:59.984456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.983922 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"kube-root-ca.crt\"" Nov 01 22:16:59.984635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.984617 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"default-dockercfg-xzpjt\"" Nov 01 22:16:59.987403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.985235 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"openshift-service-ca.crt\"" Nov 01 22:16:59.987403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:16:59.985382 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"auth-test-model-1783e3-kserve-self-signed-certs\"" Nov 01 22:17:00.029948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.029326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:00.091104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091015 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2grp\" (UniqueName: \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.091104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091082 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.091352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.091352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091314 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.091525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.091525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.091473 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.176543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.176492 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:00.195627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.195584 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2grp\" (UniqueName: \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.196212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.196181 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.196379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.196364 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.196532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.196515 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.196700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.196684 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.196846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.196832 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.198145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.198110 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.198567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.198546 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.198928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.198909 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.209092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.209067 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.221468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.221122 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2grp\" (UniqueName: \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.224467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.224297 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\") pod \"auth-test-model-1783e3-kserve-f7cb94fbf-st4md\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.242280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.242248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" event={"ID":"036d384d-5abc-4b20-a40c-090e88539d21","Type":"ContainerDied","Data":"c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332"} Nov 01 22:17:00.242508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.242488 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92036aa2f4b9180235d6efe1a26391fb70363a6bfab51ebbda151db8b0ce332" Nov 01 22:17:00.242686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.242672 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r" Nov 01 22:17:00.336964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.336922 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:00.420467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.419437 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:17:00.459694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.459098 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5628ebf8-09d6-4d65-8346-093194994d8b" path="/var/lib/kubelet/pods/5628ebf8-09d6-4d65-8346-093194994d8b/volumes" Nov 01 22:17:00.460373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.460350 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d581ae5-b177-4a34-b112-f612ba985df8" path="/var/lib/kubelet/pods/5d581ae5-b177-4a34-b112-f612ba985df8/volumes" Nov 01 22:17:00.461231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.461210 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b6afa6-2296-4256-9802-3bdc72f29da7" path="/var/lib/kubelet/pods/f8b6afa6-2296-4256-9802-3bdc72f29da7/volumes" Nov 01 22:17:00.750052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.750017 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:17:00.968653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:00.967796 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:01.275118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:01.275075 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" event={"ID":"9532214b-a852-4158-8d4b-6543bd7cac1d","Type":"ContainerStarted","Data":"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127"} Nov 01 22:17:01.275601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:01.275128 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" event={"ID":"9532214b-a852-4158-8d4b-6543bd7cac1d","Type":"ContainerStarted","Data":"7c131a147186ca98e2dc51b61fd9d745a2b07ec8ed7ddb005c5aa436c24d9cde"} Nov 01 22:17:01.296066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:01.296019 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podStartSLOduration=2.295980716 podStartE2EDuration="2.295980716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:01.295087233 +0000 UTC m=+1974.988067720" watchObservedRunningTime="2026-11-01 22:17:01.295980716 +0000 UTC m=+1974.988961202" Nov 01 22:17:01.344730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:01.344693 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:01.727885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:01.727794 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:02.517165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:02.517126 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:03.252846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:03.252809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:03.383921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:03.383882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:04.027518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:04.027479 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:04.802424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:04.802378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:05.544853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:05.544813 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:06.333248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:06.333168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:06.459482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:06.459405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:06.599596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:06.599488 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:07.269986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:07.269949 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:08.053159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:08.053123 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:08.821180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:08.821135 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:09.516831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:09.516787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:10.246102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:10.246062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:10.340905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:10.340874 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:10.353685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:10.353656 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:10.400865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:10.400835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:10.830976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:10.830932 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" containerID="cri-o://1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171" gracePeriod=45 Nov 01 22:17:11.258425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.258387 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:11.357076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.357048 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:17:11.407484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.406591 5036 generic.go:356] "Generic (PLEG): container finished" podID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerID="1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171" exitCode=0 Nov 01 22:17:11.408420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.407875 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" Nov 01 22:17:11.408420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.408248 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" event={"ID":"88166992-e75c-43a0-86a5-49e3eec8ed60","Type":"ContainerDied","Data":"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171"} Nov 01 22:17:11.408420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.408285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf" event={"ID":"88166992-e75c-43a0-86a5-49e3eec8ed60","Type":"ContainerDied","Data":"52070f59a1d1b8dd558ce06028036c4c6682079ec2f8f9a0a989422bb5c11eb9"} Nov 01 22:17:11.408420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.408307 5036 scope.go:122] "RemoveContainer" containerID="1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171" Nov 01 22:17:11.425595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.425527 5036 scope.go:122] "RemoveContainer" containerID="1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171" Nov 01 22:17:11.425872 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:11.425851 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171\": container with ID starting with 1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171 not found: ID does not exist" containerID="1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171" Nov 01 22:17:11.425959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.425884 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171"} err="failed to get container status \"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171\": rpc error: code = NotFound desc = could not find container \"1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171\": container with ID starting with 1d0b6bb36e4426bd11c2cba0174c845f98d86f21034c66e3e7035c6fc24c4171 not found: ID does not exist" Nov 01 22:17:11.498199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498165 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.498380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498286 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\" (UniqueName: \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.498380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.498555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498499 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:11.498628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498605 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:11.498691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.498768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498752 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\" (UniqueName: \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.498831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.498797 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\") pod \"88166992-e75c-43a0-86a5-49e3eec8ed60\" (UID: \"88166992-e75c-43a0-86a5-49e3eec8ed60\") " Nov 01 22:17:11.499511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.499191 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:11.499671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.499644 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.499758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.499669 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.499758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.499684 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.501195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.501165 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:17:11.501319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.501301 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:11.501777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.501754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv" pod "88166992-e75c-43a0-86a5-49e3eec8ed60" (UID: "88166992-e75c-43a0-86a5-49e3eec8ed60"). InnerVolumeSpecName "kube-api-access-2k5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:11.600929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.600885 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88166992-e75c-43a0-86a5-49e3eec8ed60-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.600929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.600921 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5cv\" (UniqueName: \"kubernetes.io/projected/88166992-e75c-43a0-86a5-49e3eec8ed60-kube-api-access-2k5cv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.601137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.600939 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88166992-e75c-43a0-86a5-49e3eec8ed60-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:11.736163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.736130 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:17:11.740599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.740573 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3e300c/test-model-3e300c-kserve-7c8854b4b7-sn7tf"] Nov 01 22:17:11.934213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:11.934172 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:12.440902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.440868 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" path="/var/lib/kubelet/pods/88166992-e75c-43a0-86a5-49e3eec8ed60/volumes" Nov 01 22:17:12.615486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.615407 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:12.819402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.818972 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:17:12.820317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.820070 5036 cpu_manager.go:414] "containerMap: removing container" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" Nov 01 22:17:12.820317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.820099 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" Nov 01 22:17:12.822556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.822463 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="88166992-e75c-43a0-86a5-49e3eec8ed60" containerName="main" Nov 01 22:17:12.833656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.828061 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.847584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.847553 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-de0cba\"/\"auth-test-model-de0cba-kserve-self-signed-certs\"" Nov 01 22:17:12.848241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.848217 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-de0cba\"/\"kube-root-ca.crt\"" Nov 01 22:17:12.848717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.848700 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-de0cba\"/\"openshift-service-ca.crt\"" Nov 01 22:17:12.850043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.850022 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-de0cba\"/\"default-dockercfg-7sgfc\"" Nov 01 22:17:12.884942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.878053 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:17:12.902887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.902855 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:12.928220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928185 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmjx\" (UniqueName: \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.928376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928271 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.928376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.928520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928408 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.928637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928620 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:12.928719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:12.928706 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040549 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmjx\" (UniqueName: \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040610 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040668 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040714 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040871 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.040949 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.042467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.041974 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.043316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.042663 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.043316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.042934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.060547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.059229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.072514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.071681 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.109469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.109300 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmjx\" (UniqueName: \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\") pod \"auth-test-model-de0cba-kserve-8456974769-g58dg\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.191510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.184610 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:13.420922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.420828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:13.532554 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:17:13.532521 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod678519b4_d469_466d_93c7_8a668a4c2b30.slice/crio-0af0fc619b03d2b18d52b21a1dc6cbd7c4040498db5717f4da026056f0a4a9dc WatchSource:0}: Error finding container 0af0fc619b03d2b18d52b21a1dc6cbd7c4040498db5717f4da026056f0a4a9dc: Status 404 returned error can't find the container with id 0af0fc619b03d2b18d52b21a1dc6cbd7c4040498db5717f4da026056f0a4a9dc Nov 01 22:17:13.535284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:13.535254 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:17:14.063510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.062961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:14.354523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.354242 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:14.510844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.510777 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:14.519638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.519254 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r"] Nov 01 22:17:14.519638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.519299 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" event={"ID":"678519b4-d469-466d-93c7-8a668a4c2b30","Type":"ContainerStarted","Data":"55aa95234df2c001291f733f3baf3dad3e57ce2cc2b1fc15a1711b16a23374ee"} Nov 01 22:17:14.519638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.519326 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" event={"ID":"678519b4-d469-466d-93c7-8a668a4c2b30","Type":"ContainerStarted","Data":"0af0fc619b03d2b18d52b21a1dc6cbd7c4040498db5717f4da026056f0a4a9dc"} Nov 01 22:17:14.528467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.525679 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-487639-mrl5r"] Nov 01 22:17:14.551463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.549534 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podStartSLOduration=2.549509777 podStartE2EDuration="2.549509777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:14.548294723 +0000 UTC m=+1988.241275210" watchObservedRunningTime="2026-11-01 22:17:14.549509777 +0000 UTC m=+1988.242490263" Nov 01 22:17:14.634542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.634409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:14.825496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.822498 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:14.952192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:14.952094 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:15.152316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:15.152274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:15.185627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:15.185592 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:15.314055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:15.314022 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:15.595533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:15.574529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:15.968116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:15.968013 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:16.441840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.441803 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036d384d-5abc-4b20-a40c-090e88539d21" path="/var/lib/kubelet/pods/036d384d-5abc-4b20-a40c-090e88539d21/volumes" Nov 01 22:17:16.740843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.740747 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:17:16.750421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.750288 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.759953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.759922 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"kube-root-ca.crt\"" Nov 01 22:17:16.760337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.760315 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"openshift-service-ca.crt\"" Nov 01 22:17:16.761423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.761394 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:17:16.816484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.816360 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"auth-test-model-e2bf63-kserve-self-signed-certs\"" Nov 01 22:17:16.820055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.820033 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"default-dockercfg-rxnds\"" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph6r\" (UniqueName: \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893579 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893720 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.899813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.893810 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.931565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.931053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:16.995770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.995680 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.996099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.996076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.996359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.996319 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph6r\" (UniqueName: \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.996534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.996513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.996823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.996808 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.996995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.996981 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.997608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.997580 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.998014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.997992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:16.998965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:16.998945 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:17.012543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.009699 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:17.012543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.011960 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:17.014028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.013998 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph6r\" (UniqueName: \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\") pod \"auth-test-model-e2bf63-kserve-775f7f777-ggvc6\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:17.097472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.096546 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:17.099049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.097809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:17.557309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.557273 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:17:17.694584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.694543 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" event={"ID":"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5","Type":"ContainerStarted","Data":"8ce02d4d6f513510ca91fbbf9dd1f6f30968c078d2a742c33cf646f3465dfda5"} Nov 01 22:17:17.694748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.694595 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" event={"ID":"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5","Type":"ContainerStarted","Data":"a35f8e27a93b089d13d38660bc722802ea6fbbeafdb6ef6cf3c9f6e1eef3ace0"} Nov 01 22:17:17.725986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.725926 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podStartSLOduration=1.725908923 podStartE2EDuration="1.725908923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:17.714173461 +0000 UTC m=+1991.407153949" watchObservedRunningTime="2026-11-01 22:17:17.725908923 +0000 UTC m=+1991.418889410" Nov 01 22:17:17.848681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:17.845493 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:18.097621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:18.097587 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:18.405941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:18.405900 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:18.536776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:18.536735 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:18.713336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:18.713248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:18.864696 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:18.864655 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:17:19.454766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:19.454727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:19.672434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:19.672390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:19.804470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:19.801040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:20.538425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:20.538379 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:21.401759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:21.401718 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:17:21.465891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:21.465850 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:22.179970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.179922 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:22.310750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.310660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:22.489492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.486123 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:17:22.490080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.490056 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:17:22.490243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.490231 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:17:22.490366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.490289 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" containerID="cri-o://5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264" gracePeriod=30 Nov 01 22:17:22.490613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.490586 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" containerID="cri-o://d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0" gracePeriod=30 Nov 01 22:17:22.491186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.490681 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" containerID="cri-o://4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b" gracePeriod=30 Nov 01 22:17:22.530155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:22.530109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:23.187872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.187835 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:23.515821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.515486 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72"] Nov 01 22:17:23.533956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.533923 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:23.534142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.534028 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.534385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.534349 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72"] Nov 01 22:17:23.539778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.539751 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:17:23.540854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.540834 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:17:23.674125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.674070 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.674583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.674553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8q7\" (UniqueName: \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.675042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.675019 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.805775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.805728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.809182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.809145 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8q7\" (UniqueName: \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.809992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.809964 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.810863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.806802 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.847616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.847581 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8q7\" (UniqueName: \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.852654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.847875 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.859735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.859709 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:23.960468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:23.958283 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:24.182558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:24.174981 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:24.346647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:24.346610 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72"] Nov 01 22:17:24.951436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:24.951394 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" event={"ID":"04bab3c2-0995-4a08-aa35-0e9aa3f0d486","Type":"ContainerStarted","Data":"9036344dda94c6dfbdc085117f52bfd6fc5a96e2ffbda9921637743819ecebdc"} Nov 01 22:17:24.951911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:24.951467 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" event={"ID":"04bab3c2-0995-4a08-aa35-0e9aa3f0d486","Type":"ContainerStarted","Data":"2ddde3504c0880c0cf5bd552bece2d7cadba3a47835ed58e8a6a965fe54a7736"} Nov 01 22:17:24.993466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:24.990901 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" podStartSLOduration=1.990883405 podStartE2EDuration="1.990883405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:24.970798984 +0000 UTC m=+1998.663779471" watchObservedRunningTime="2026-11-01 22:17:24.990883405 +0000 UTC m=+1998.683863896" Nov 01 22:17:25.163230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:25.163141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:25.971991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:25.971947 5036 generic.go:356] "Generic (PLEG): container finished" podID="04bab3c2-0995-4a08-aa35-0e9aa3f0d486" containerID="9036344dda94c6dfbdc085117f52bfd6fc5a96e2ffbda9921637743819ecebdc" exitCode=0 Nov 01 22:17:25.972548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:25.972298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" event={"ID":"04bab3c2-0995-4a08-aa35-0e9aa3f0d486","Type":"ContainerDied","Data":"9036344dda94c6dfbdc085117f52bfd6fc5a96e2ffbda9921637743819ecebdc"} Nov 01 22:17:26.199604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:26.191980 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:27.107653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.107623 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:27.114120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.113801 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:27.114120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.113852 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:27.133303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.133278 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:27.176312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.175921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:27.429791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.429760 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:27.553592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.552939 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\") pod \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " Nov 01 22:17:27.553592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.553011 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\") pod \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " Nov 01 22:17:27.553592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.553133 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\" (UniqueName: \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\") pod \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\" (UID: \"04bab3c2-0995-4a08-aa35-0e9aa3f0d486\") " Nov 01 22:17:27.554626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.554601 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca" pod "04bab3c2-0995-4a08-aa35-0e9aa3f0d486" (UID: "04bab3c2-0995-4a08-aa35-0e9aa3f0d486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:27.559568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.559537 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca" pod "04bab3c2-0995-4a08-aa35-0e9aa3f0d486" (UID: "04bab3c2-0995-4a08-aa35-0e9aa3f0d486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:17:27.566731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.566687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7" pod "04bab3c2-0995-4a08-aa35-0e9aa3f0d486" (UID: "04bab3c2-0995-4a08-aa35-0e9aa3f0d486"). InnerVolumeSpecName "kube-api-access-7v8q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:27.659491 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:27.659436 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8a94d1_7827_40e9_ae5f_139ac8b63092.slice/crio-5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f901f6_dc11_49bf_acdc_72cae5c2c9f7.slice/crio-d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b7ab59_79fa_4c9f_8356_bcbb1118eb20.slice/crio-4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:17:27.662575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.659670 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:27.662575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.659723 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:27.662575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.659740 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8q7\" (UniqueName: \"kubernetes.io/projected/04bab3c2-0995-4a08-aa35-0e9aa3f0d486-kube-api-access-7v8q7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:27.979469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:27.976684 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:17:28.105000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.104955 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.105286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.105267 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.105421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.105406 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.105632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.105614 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.105789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.105774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.105909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.105895 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.106064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.106047 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.106231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.106217 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.106527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.106508 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\") pod \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\" (UID: \"ba8a94d1-7827-40e9-ae5f-139ac8b63092\") " Nov 01 22:17:28.122211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.122171 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.125113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.124109 5036 generic.go:356] "Generic (PLEG): container finished" podID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerID="4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b" exitCode=0 Nov 01 22:17:28.125113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.124317 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" event={"ID":"68b7ab59-79fa-4c9f-8356-bcbb1118eb20","Type":"ContainerDied","Data":"4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b"} Nov 01 22:17:28.126320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.125924 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.127700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.127664 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:28.127879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.127859 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.128394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.128372 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:28.157552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.133881 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.157552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.133960 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" event={"ID":"04bab3c2-0995-4a08-aa35-0e9aa3f0d486","Type":"ContainerDied","Data":"2ddde3504c0880c0cf5bd552bece2d7cadba3a47835ed58e8a6a965fe54a7736"} Nov 01 22:17:28.157552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.142733 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ddde3504c0880c0cf5bd552bece2d7cadba3a47835ed58e8a6a965fe54a7736" Nov 01 22:17:28.157552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.134042 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72" Nov 01 22:17:28.157552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.156115 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.157929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.157573 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:28.192080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.173067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.192080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.173476 5036 generic.go:356] "Generic (PLEG): container finished" podID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerID="d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0" exitCode=0 Nov 01 22:17:28.192080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.173592 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" event={"ID":"57f901f6-dc11-49bf-acdc-72cae5c2c9f7","Type":"ContainerDied","Data":"d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0"} Nov 01 22:17:28.192354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.192210 5036 generic.go:356] "Generic (PLEG): container finished" podID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerID="5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264" exitCode=0 Nov 01 22:17:28.193578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.192541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerDied","Data":"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264"} Nov 01 22:17:28.193578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.192586 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" event={"ID":"ba8a94d1-7827-40e9-ae5f-139ac8b63092","Type":"ContainerDied","Data":"430f7dbd692b4377e849ee7e9176659c3b7dde34a67237eeb079164226126180"} Nov 01 22:17:28.193578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.192617 5036 scope.go:122] "RemoveContainer" containerID="5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264" Nov 01 22:17:28.193578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.193329 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8" Nov 01 22:17:28.205816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.205794 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:28.244587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244550 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244590 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244610 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244625 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244639 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244653 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244669 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a94d1-7827-40e9-ae5f-139ac8b63092-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.244774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.244683 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a94d1-7827-40e9-ae5f-139ac8b63092-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.246055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.245929 5036 scope.go:122] "RemoveContainer" containerID="28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c" Nov 01 22:17:28.257539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.257510 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths" pod "ba8a94d1-7827-40e9-ae5f-139ac8b63092" (UID: "ba8a94d1-7827-40e9-ae5f-139ac8b63092"). InnerVolumeSpecName "kube-api-access-psths". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.263411 5036 scope.go:122] "RemoveContainer" containerID="5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:28.263874 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264\": container with ID starting with 5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264 not found: ID does not exist" containerID="5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.263910 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264"} err="failed to get container status \"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264\": rpc error: code = NotFound desc = could not find container \"5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264\": container with ID starting with 5f7d4ce8dca543a09daab0ecb250ead99b2aeb5bbe91ebc908c5011840b00264 not found: ID does not exist" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.263935 5036 scope.go:122] "RemoveContainer" containerID="28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:28.264258 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c\": container with ID starting with 28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c not found: ID does not exist" containerID="28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c" Nov 01 22:17:28.264623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.264287 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c"} err="failed to get container status \"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c\": rpc error: code = NotFound desc = could not find container \"28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c\": container with ID starting with 28e3f75b17519f86836b705b92afb211bc718a0f12dfc57aa54aa69d7207160c not found: ID does not exist" Nov 01 22:17:28.277311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.277288 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.356843 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.356951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.356983 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357046 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357088 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357243 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357287 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357376 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.357438 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\") pod \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\" (UID: \"68b7ab59-79fa-4c9f-8356-bcbb1118eb20\") " Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.358433 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psths\" (UniqueName: \"kubernetes.io/projected/ba8a94d1-7827-40e9-ae5f-139ac8b63092-kube-api-access-psths\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.359018 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.359431 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.363139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.359823 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.369678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.366523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.369678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.367033 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.369678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.367083 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:28.399464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.383830 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:28.428658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.426108 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.443997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.434134 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd" pod "68b7ab59-79fa-4c9f-8356-bcbb1118eb20" (UID: "68b7ab59-79fa-4c9f-8356-bcbb1118eb20"). InnerVolumeSpecName "kube-api-access-l4mjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.460160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.460130 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.467505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.460642 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:17:28.468083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.468059 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471258 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471292 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471317 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mjd\" (UniqueName: \"kubernetes.io/projected/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-kube-api-access-l4mjd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471333 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471348 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471362 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.471404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.471380 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b7ab59-79fa-4c9f-8356-bcbb1118eb20-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.572478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.572407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.572621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.572527 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.572873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.572849 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.572964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.572914 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.573063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.573029 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.573165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.573109 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.573239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.573207 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.573334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.573318 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.573573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.573534 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\") pod \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\" (UID: \"57f901f6-dc11-49bf-acdc-72cae5c2c9f7\") " Nov 01 22:17:28.575324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.575295 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:17:28.575634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.575591 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.575809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.575725 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:28.576157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.576131 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.580547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.580522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.582315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.582291 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.592520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.592496 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.592671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.592655 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.592781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.592768 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.595513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.595489 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.632472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.626654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "kube-api-access-csp8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.638226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.632741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:28.638226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.637146 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-64bw8"] Nov 01 22:17:28.638226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.637414 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:28.638226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.637726 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token" pod "57f901f6-dc11-49bf-acdc-72cae5c2c9f7" (UID: "57f901f6-dc11-49bf-acdc-72cae5c2c9f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:28.701541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.700276 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.701541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.700319 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csp8p\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-kube-api-access-csp8p\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.701541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.700337 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.701541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.700352 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.701541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.700369 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f901f6-dc11-49bf-acdc-72cae5c2c9f7-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:28.847560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:28.847520 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:29.204755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.204726 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" Nov 01 22:17:29.204755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.204740 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg" event={"ID":"57f901f6-dc11-49bf-acdc-72cae5c2c9f7","Type":"ContainerDied","Data":"da67fdb44603759318437f354891b512f429f36354dc8b2fadc3dec9377f9752"} Nov 01 22:17:29.205299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.204807 5036 scope.go:122] "RemoveContainer" containerID="d7ed0d82a4b1ecd0bfeda4b482f172c110cb487d0b8f98899af3db53774ed7d0" Nov 01 22:17:29.215527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.215482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" event={"ID":"68b7ab59-79fa-4c9f-8356-bcbb1118eb20","Type":"ContainerDied","Data":"8eded9dc560d9812a4ba767b44dff6c69f9438c4aeacdcdd504c9aecec1b519b"} Nov 01 22:17:29.215700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.215684 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" Nov 01 22:17:29.228089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.228065 5036 scope.go:122] "RemoveContainer" containerID="4d0224ddd83297e56884e169362520d7e127dd5b7fc63c97f025e4c4ef17b22b" Nov 01 22:17:29.246320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.246286 5036 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:17:29.246545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.246368 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:29.249921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.249894 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:17:29.280867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.280834 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-ctbrg"] Nov 01 22:17:29.310378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.310342 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:17:29.328320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.328285 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-7jtfw"] Nov 01 22:17:29.622720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:29.622681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:30.392293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:30.392253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:30.453596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:30.453516 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" path="/var/lib/kubelet/pods/57f901f6-dc11-49bf-acdc-72cae5c2c9f7/volumes" Nov 01 22:17:30.454492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:30.454432 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" path="/var/lib/kubelet/pods/68b7ab59-79fa-4c9f-8356-bcbb1118eb20/volumes" Nov 01 22:17:30.455232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:30.455175 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" path="/var/lib/kubelet/pods/ba8a94d1-7827-40e9-ae5f-139ac8b63092/volumes" Nov 01 22:17:31.348388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:31.348348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:32.011273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:32.011236 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:32.663854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:32.663812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:33.429590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:33.429540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:33.649565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:33.648724 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:33.789471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:33.788373 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:34.276512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.276475 5036 generic.go:356] "Generic (PLEG): container finished" podID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerID="9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313" exitCode=0 Nov 01 22:17:34.276821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.276612 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerDied","Data":"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313"} Nov 01 22:17:34.277115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.277095 5036 scope.go:122] "RemoveContainer" containerID="9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313" Nov 01 22:17:34.281091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.279430 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:34.281091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.279569 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:34.281091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.279625 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:34.434460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:34.434405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:35.137048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:35.137014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:35.324492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:35.322279 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerStarted","Data":"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0"} Nov 01 22:17:35.574893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:35.574857 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:17:35.576121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:35.576094 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:35.576245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:35.576153 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:36.023152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:36.023109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:36.575662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:36.575618 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:36.576057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:36.575681 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:36.948596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:36.948497 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:37.576731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:37.576699 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:37.577283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:37.577251 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:37.662488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:37.662377 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:37.811323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:37.811249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:38.575673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:38.575632 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:38.575883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:38.575695 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:38.611133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:38.611093 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:39.233850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.233802 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:39.513522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.508815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:39.615475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.580872 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:39.615475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.580946 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:39.752467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.751276 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:17:39.764604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.764511 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72"] Nov 01 22:17:39.775482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.775213 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:39.777053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.776898 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3e300c-dnk72"] Nov 01 22:17:39.916075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:39.916038 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:40.223042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:40.222986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:40.440765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:40.440726 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bab3c2-0995-4a08-aa35-0e9aa3f0d486" path="/var/lib/kubelet/pods/04bab3c2-0995-4a08-aa35-0e9aa3f0d486/volumes" Nov 01 22:17:40.575560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:40.575530 5036 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:17:40.575747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:40.575591 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 01 22:17:40.711201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:40.710739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:41.536553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:41.536512 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:41.583683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:41.583436 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:17:41.593492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:41.590291 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:17:42.340883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:42.340844 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:43.023433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:43.023398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:43.712201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:43.712131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:44.455708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:44.455354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:45.000728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.000688 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:17:45.001812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001762 5036 cpu_manager.go:414] "containerMap: removing container" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" Nov 01 22:17:45.001812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001790 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" Nov 01 22:17:45.001812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001813 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001821 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001834 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001842 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001866 5036 cpu_manager.go:414] "containerMap: removing container" podUID="04bab3c2-0995-4a08-aa35-0e9aa3f0d486" containerName="revoke-keys" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001875 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bab3c2-0995-4a08-aa35-0e9aa3f0d486" containerName="revoke-keys" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001926 5036 cpu_manager.go:414] "containerMap: removing container" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" Nov 01 22:17:45.002055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.001934 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" Nov 01 22:17:45.002280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.002129 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b7ab59-79fa-4c9f-8356-bcbb1118eb20" containerName="istio-proxy" Nov 01 22:17:45.002280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.002143 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.002155 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8a94d1-7827-40e9-ae5f-139ac8b63092" containerName="istio-proxy" Nov 01 22:17:45.002280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.002166 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bab3c2-0995-4a08-aa35-0e9aa3f0d486" containerName="revoke-keys" Nov 01 22:17:45.002280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.002184 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f901f6-dc11-49bf-acdc-72cae5c2c9f7" containerName="istio-proxy" Nov 01 22:17:45.008375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.008350 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.013605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.013576 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1783e3-openshift-default-dockercfg-kwmww\"" Nov 01 22:17:45.045079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.043486 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:17:45.100219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100226 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4wt\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100696 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100771 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.100924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.100837 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.101072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.101055 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.175520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.172992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:45.196691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.196651 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:17:45.203666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.203157 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.203666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.203640 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.203691 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204285 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4wt\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204489 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204530 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204558 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204595 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.204818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.204650 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.222947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.212156 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.222947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.212673 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.222947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.213283 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.250423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.244891 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.255943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.255856 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:17:45.256153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.256001 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.258872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.258842 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4wt\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.260394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.260339 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.261880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.261858 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.262612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.262589 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e3890e-openshift-default-dockercfg-j98nz\"" Nov 01 22:17:45.338286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.338259 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:45.421725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.421693 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.421857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.421779 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.421901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.421866 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422010 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422048 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422098 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422191 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k255\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422232 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.422334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.422319 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524137 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524258 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524468 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524650 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524711 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524744 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.524852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524871 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k255\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524921 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.524983 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.525014 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.525204 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.525773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.525748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.527592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.527564 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.527743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.527726 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.536690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.536664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.537052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.537036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k255\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.613080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.613054 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:45.638523 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:17:45.638495 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fb4333_23a5_4f4a_aa3b_069ef0074ba6.slice/crio-5005266c19676b2c7915c10273d7590458f7fa4cc84cf5dece9010603d6689d2 WatchSource:0}: Error finding container 5005266c19676b2c7915c10273d7590458f7fa4cc84cf5dece9010603d6689d2: Status 404 returned error can't find the container with id 5005266c19676b2c7915c10273d7590458f7fa4cc84cf5dece9010603d6689d2 Nov 01 22:17:45.640589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.640481 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:17:45.641682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.641650 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:45.641764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.641713 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:45.641764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.641742 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:45.653434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.653407 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:17:45.662115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.660899 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.665278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.664840 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d6e25f-openshift-default-dockercfg-t9xvc\"" Nov 01 22:17:45.672102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.672078 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:17:45.692963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.692926 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" event={"ID":"26fb4333-23a5-4f4a-aa3b-069ef0074ba6","Type":"ContainerStarted","Data":"5005266c19676b2c7915c10273d7590458f7fa4cc84cf5dece9010603d6689d2"} Nov 01 22:17:45.728315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728284 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.728510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.728599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728579 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.728704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728658 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.728873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728855 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.728958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.728943 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.729070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.729053 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbssr\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.729180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.729163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.729239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.729221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831307 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831374 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831599 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831633 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbssr\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831704 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831760 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831831 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.831935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.831886 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.832585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.832552 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.832842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.832820 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.832929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.832869 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.833095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.833072 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.833343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.833319 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.834955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.834934 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.835289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.835272 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.845058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.845036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.845787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.845768 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbssr\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:45.921737 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:17:45.921707 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76cc9f5_fbd4_41b1_83b1_c9a68bd1d93d.slice/crio-dccaa351ffeb542fcf2ccb2562518aa4468b6c258aa62a7f94dda52a89d10c7e WatchSource:0}: Error finding container dccaa351ffeb542fcf2ccb2562518aa4468b6c258aa62a7f94dda52a89d10c7e: Status 404 returned error can't find the container with id dccaa351ffeb542fcf2ccb2562518aa4468b6c258aa62a7f94dda52a89d10c7e Nov 01 22:17:45.922353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.922327 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:17:45.925452 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.925405 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:45.925534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.925501 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:45.925578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.925535 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:46.005592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:45.995139 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:46.044479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.044223 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:46.386871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.386815 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:46.387003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.386984 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:46.387095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.387072 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:17:46.394933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.392065 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:17:46.719354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.719258 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" event={"ID":"9641db1e-73b2-482b-b72e-c05903aaed8d","Type":"ContainerStarted","Data":"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976"} Nov 01 22:17:46.719661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.719628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" event={"ID":"9641db1e-73b2-482b-b72e-c05903aaed8d","Type":"ContainerStarted","Data":"7d7dfa576b5c3682fb527a2790fee13ef7611c90cc6715e2e14486cf1fe98331"} Nov 01 22:17:46.722652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.722622 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" event={"ID":"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d","Type":"ContainerStarted","Data":"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926"} Nov 01 22:17:46.722852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.722831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" event={"ID":"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d","Type":"ContainerStarted","Data":"dccaa351ffeb542fcf2ccb2562518aa4468b6c258aa62a7f94dda52a89d10c7e"} Nov 01 22:17:46.726268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.726227 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" event={"ID":"26fb4333-23a5-4f4a-aa3b-069ef0074ba6","Type":"ContainerStarted","Data":"b4ec59eb5560cc71fd894eb07f88a9b524679f2fd368f9aa8a897300876b48c3"} Nov 01 22:17:46.755011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.754577 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podStartSLOduration=1.7545589499999998 podStartE2EDuration="1.75455895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.753711647 +0000 UTC m=+2020.446692133" watchObservedRunningTime="2026-11-01 22:17:46.75455895 +0000 UTC m=+2020.447539431" Nov 01 22:17:46.768483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.766799 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:46.801771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:46.788384 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podStartSLOduration=2.7883609270000003 podStartE2EDuration="2.788360927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.784935375 +0000 UTC m=+2020.477915862" watchObservedRunningTime="2026-11-01 22:17:46.788360927 +0000 UTC m=+2020.481341414" Nov 01 22:17:47.004501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.004355 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podStartSLOduration=2.004331559 podStartE2EDuration="2.004331559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.964624235 +0000 UTC m=+2020.657604717" watchObservedRunningTime="2026-11-01 22:17:47.004331559 +0000 UTC m=+2020.697312046" Nov 01 22:17:47.057604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.057560 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:47.058093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.057793 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:47.058093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.057841 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:47.341068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.340905 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:47.347577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.341698 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:47.347776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.347605 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:47.541670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.541357 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:47.621299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.621208 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:47.635614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.635576 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:47.635922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.635886 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:47.687157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.687120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:47.840469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.840047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:47.888801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.888703 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:17:47.889675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:47.889633 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" containerID="cri-o://0cc20956c4426d4e6e7c9238443c9812463fd72ae00ade5184371d00abeddaae" gracePeriod=30 Nov 01 22:17:48.100292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.099357 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:17:48.100292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.099651 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" containerID="cri-o://2e0ef8af766fba809f39684f635afbba27b6c462dc661126366a9034fc510156" gracePeriod=30 Nov 01 22:17:48.171619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.153595 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:48.200078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.179256 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:48.201583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.201532 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:48.281602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.263486 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:17:48.281602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.263733 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" containerID="cri-o://c4c7fc9bed79b066e80489282e1f301176ec8d0600cbcdf4b407bc722b305604" gracePeriod=30 Nov 01 22:17:48.392517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.380195 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:17:48.392517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.380475 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" containerID="cri-o://c5db5fd3d10009344d93802d152fc6d0f860adc6d03c2ad4dade50fdf22bfdcf" gracePeriod=30 Nov 01 22:17:48.596331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.596292 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:48.596659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.596608 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:48.825466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.811500 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:48.986465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.828617 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:48.986846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.986716 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:49.002463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.865123 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:17:49.005146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.911633 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.005994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:48.863918 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:17:49.040403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.040362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.103543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.103488 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:49.105594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.105553 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:49.143391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.143341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:49.432745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.432683 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:49.433070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.433019 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:49.454558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.433992 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.712060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.711969 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:49.785965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.785921 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:49.786240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.728987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:49.786473 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:17:49.728369 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:17:49.823689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.823626 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:49.824032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.823976 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:49.824369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.824312 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.854953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.854911 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.866467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.863143 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/health\": context deadline exceeded" Nov 01 22:17:49.933471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.929034 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:17:49.953050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.953012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:49.962058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.962007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.974374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.974274 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.975085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.974629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:49.990494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.990435 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": context deadline exceeded" Nov 01 22:17:49.991529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:49.991498 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.005591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.005532 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.008277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.007891 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.009151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.008533 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:17:50.009488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.009471 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:17:50.049482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.049415 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.050478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.050199 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:17:50.051944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.051903 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.068042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.067988 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:50.068282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.068045 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:50.136367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.136312 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.295471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.288514 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.356642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.356493 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": context deadline exceeded" Nov 01 22:17:50.357154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.356884 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:17:50.366641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.366032 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:50.366641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.366098 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:50.386634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.385626 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:17:50.538358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.538223 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:17:50.539257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.539217 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.617596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.617477 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:50.617905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.617847 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:50.763468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.762935 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:17:50.832057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.832018 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.832824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.832803 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:17:50.866470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.840906 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:17:50.886268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.833385 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.944472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.942631 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:50.944472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.942709 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:50.944472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.943021 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.944472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.943423 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:17:50.958433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.957663 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": context deadline exceeded" Nov 01 22:17:50.992491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.962868 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:50.992491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.963206 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:50.994295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:50.994255 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.078477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.014610 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.078477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.078130 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:17:51.079231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.079195 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.085470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.079582 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:17:51.184270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.179753 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:51.184270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.179818 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:51.280745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.264720 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:51.280745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.265678 5036 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:51.280745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.265732 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:51.280745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.277359 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:17:51.280745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.277484 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 01 22:17:51.318331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.318247 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:17:51.343617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.339553 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:51.343617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.339612 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:51.359141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.356856 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.359141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.357147 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:17:51.359141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.358204 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.359141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.358566 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:17:51.359141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.359061 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/health\": context deadline exceeded" Nov 01 22:17:51.506349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.506258 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:17:51.558169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.558124 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.558820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.558796 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:51.619479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.619015 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:51.619479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.619072 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:51.693099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.692163 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.830484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.795524 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.869471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.849005 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.873086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.873050 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.959017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.957400 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.959017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.958746 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.959288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.959051 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:17:51.967381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.965416 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:51.971676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:51.971518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.062936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.062898 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:52.063239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.063207 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:52.066866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.066840 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:17:52.072920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.069111 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.072920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.072267 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:52.092116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.092000 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.126876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.126832 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": context deadline exceeded" Nov 01 22:17:52.137620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.137577 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.281815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.281745 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": context deadline exceeded" Nov 01 22:17:52.336009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.335978 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:17:52.339868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.339846 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:17:52.343300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.343240 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:52.352887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.352860 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:52.357547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.357518 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": context deadline exceeded" Nov 01 22:17:52.359168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.358690 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.447831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.447785 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:52.727523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.727325 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:52.727523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.727409 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:52.803487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.803122 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:17:52.979010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.975097 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:17:52.979010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.975519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:17:52.979010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.976147 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:17:52.979010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.977639 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:17:53.020922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:52.987888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:53.046469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.031840 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:17:53.626158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.047931 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": context deadline exceeded" Nov 01 22:17:53.655469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.113897 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": context deadline exceeded" Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.570728 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.582960 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.850646 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.689258 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.629838 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.620561 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:53.856508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.851023 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:53.902284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.902173 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Nov 01 22:17:53.928496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.907237 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Nov 01 22:17:53.928496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.906848 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/health\": context deadline exceeded" Nov 01 22:17:53.932571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.932524 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 01 22:17:53.967615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:53.967550 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:54.049556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.027115 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:17:54.096706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.091082 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:54.121486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.116007 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": context deadline exceeded" Nov 01 22:17:54.129490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.127482 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:54.185543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.178115 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:54.185543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.178176 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:54.416479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.393527 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": context deadline exceeded" Nov 01 22:17:54.423941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.423905 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 01 22:17:54.424180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.424148 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 01 22:17:54.584665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.573843 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:54.683297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.682708 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": context deadline exceeded" Nov 01 22:17:54.707744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.701679 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/health\": context deadline exceeded" Nov 01 22:17:54.725072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.713142 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:17:54.725072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.714053 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": context deadline exceeded" Nov 01 22:17:54.725072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.715302 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:17:54.725072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.719362 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 01 22:17:54.796675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.796614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:54.860954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.854611 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:17:54.861496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.857239 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": context deadline exceeded" Nov 01 22:17:54.999477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:54.995733 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.094885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.094841 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.098615 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.098690 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.111096 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.116237 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.368894 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 01 22:17:55.368960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.226970 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:17:55.369540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.201213 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.370334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.291589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:55.370886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.357087 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.370886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.356800 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.394277 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.398977 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399069 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399114 5036 generic.go:356] "Generic (PLEG): container finished" podID="02428bf4-f476-4f4d-9847-5668c194869f" containerID="c5db5fd3d10009344d93802d152fc6d0f860adc6d03c2ad4dade50fdf22bfdcf" exitCode=0 Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399174 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399248 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399278 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" event={"ID":"02428bf4-f476-4f4d-9847-5668c194869f","Type":"ContainerDied","Data":"c5db5fd3d10009344d93802d152fc6d0f860adc6d03c2ad4dade50fdf22bfdcf"} Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399465 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:17:55.402015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.399522 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.406237 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.407154 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.410133 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.410186 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.417567 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.418410 5036 generic.go:356] "Generic (PLEG): container finished" podID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerID="c4c7fc9bed79b066e80489282e1f301176ec8d0600cbcdf4b407bc722b305604" exitCode=0 Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.418679 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" event={"ID":"52e28a5c-aaaa-4698-9c75-62d149c5c19b","Type":"ContainerDied","Data":"c4c7fc9bed79b066e80489282e1f301176ec8d0600cbcdf4b407bc722b305604"} Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.418740 5036 scope.go:122] "RemoveContainer" containerID="c4c7fc9bed79b066e80489282e1f301176ec8d0600cbcdf4b407bc722b305604" Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.424506 5036 generic.go:356] "Generic (PLEG): container finished" podID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerID="2e0ef8af766fba809f39684f635afbba27b6c462dc661126366a9034fc510156" exitCode=0 Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.424907 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" event={"ID":"6fb8b132-d7f0-4db8-8fc4-34b424107dd2","Type":"ContainerDied","Data":"2e0ef8af766fba809f39684f635afbba27b6c462dc661126366a9034fc510156"} Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.433078 5036 generic.go:356] "Generic (PLEG): container finished" podID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerID="0cc20956c4426d4e6e7c9238443c9812463fd72ae00ade5184371d00abeddaae" exitCode=0 Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.433478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" event={"ID":"12da8b2d-cbec-446a-957a-2bd7eae29773","Type":"ContainerDied","Data":"0cc20956c4426d4e6e7c9238443c9812463fd72ae00ade5184371d00abeddaae"} Nov 01 22:17:55.435059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.433725 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf" Nov 01 22:17:55.486664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.486635 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:17:55.493555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.493331 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:17:55.529535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.506116 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:17:55.556667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.556639 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:55.558791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.557737 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:55.558791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.557812 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:17:55.582804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.581221 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.583172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.583149 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.583314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.583298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.583631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.583605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.583926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.583905 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.584114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.584087 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.584311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.584293 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.584469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.584438 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.584597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.584578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.599300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.599266 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:55.601098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.601071 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.601599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.601575 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:55.607697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.606057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.609403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.606256 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.609591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.608135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:55.610403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.606726 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:17:55.611057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.604043 5036 scope.go:122] "RemoveContainer" containerID="0cc20956c4426d4e6e7c9238443c9812463fd72ae00ade5184371d00abeddaae" Nov 01 22:17:55.611257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.610384 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.612933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.612895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:55.614683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.614663 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:17:55.614851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.614837 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:17:55.615251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.614977 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:17:55.615435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.615420 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:17:55.615610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.615595 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:17:55.615754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.615741 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:17:55.616764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.616741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.617058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.617032 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.618252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.618215 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:55.736363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.736315 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.737289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.737238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.737554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.737537 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.737955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.737936 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.738231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.738201 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.738421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.738395 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.738528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.738511 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\") pod \"12da8b2d-cbec-446a-957a-2bd7eae29773\" (UID: \"12da8b2d-cbec-446a-957a-2bd7eae29773\") " Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.738686 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.738766 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\") pod \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\" (UID: \"52e28a5c-aaaa-4698-9c75-62d149c5c19b\") " Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739619 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739642 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da8b2d-cbec-446a-957a-2bd7eae29773-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739657 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739673 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739687 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739700 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739754 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.739774 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.740396 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.740608 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.740771 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.742140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.740930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.760775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.745902 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:17:55.760775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.757619 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.760775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.758032 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "kube-api-access-27hz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.761695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.761666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.764592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.764573 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:17:55.765021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.764960 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:17:55.769141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.769069 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz" pod "12da8b2d-cbec-446a-957a-2bd7eae29773" (UID: "12da8b2d-cbec-446a-957a-2bd7eae29773"). InnerVolumeSpecName "kube-api-access-7xbqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.771787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.771565 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token" pod "52e28a5c-aaaa-4698-9c75-62d149c5c19b" (UID: "52e28a5c-aaaa-4698-9c75-62d149c5c19b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.796527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.796437 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:17:55.842187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.840914 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.842497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.842470 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.842668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.842650 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.842880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.842861 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.843063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843042 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.843220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843203 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.843363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843347 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.843520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843491 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.843659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843643 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.843834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.843808 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.844068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844050 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.844242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844208 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.844413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844396 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.844560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844544 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\") pod \"02428bf4-f476-4f4d-9847-5668c194869f\" (UID: \"02428bf4-f476-4f4d-9847-5668c194869f\") " Nov 01 22:17:55.844708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844691 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.844839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.844967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.845196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.845177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\") pod \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\" (UID: \"6fb8b132-d7f0-4db8-8fc4-34b424107dd2\") " Nov 01 22:17:55.846156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.846117 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.846297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.846279 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.871952 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.871981 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da8b2d-cbec-446a-957a-2bd7eae29773-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.871996 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.872011 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.872025 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbqz\" (UniqueName: \"kubernetes.io/projected/12da8b2d-cbec-446a-957a-2bd7eae29773-kube-api-access-7xbqz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.872040 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e28a5c-aaaa-4698-9c75-62d149c5c19b-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.876037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.872055 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hz7\" (UniqueName: \"kubernetes.io/projected/52e28a5c-aaaa-4698-9c75-62d149c5c19b-kube-api-access-27hz7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844403 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.844732 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.846747 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.847126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.857893 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.858082 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.872268 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.876603 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.876801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.890534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.890480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.927680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.927623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.928558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.927987 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:55.928558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.928117 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "kube-api-access-6922n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.928558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.928193 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "kube-api-access-xh7tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.928558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.928253 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:55.928558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.928521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:55.929400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.929358 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy" pod "6fb8b132-d7f0-4db8-8fc4-34b424107dd2" (UID: "6fb8b132-d7f0-4db8-8fc4-34b424107dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.930062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.930039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy" pod "02428bf4-f476-4f4d-9847-5668c194869f" (UID: "02428bf4-f476-4f4d-9847-5668c194869f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975219 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02428bf4-f476-4f4d-9847-5668c194869f-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975250 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975260 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975269 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975278 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975286 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975296 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975305 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6922n\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-kube-api-access-6922n\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975314 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02428bf4-f476-4f4d-9847-5668c194869f-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975328 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975337 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975345 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975354 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7tz\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-kube-api-access-xh7tz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975363 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975372 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02428bf4-f476-4f4d-9847-5668c194869f-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975380 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975387 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02428bf4-f476-4f4d-9847-5668c194869f-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:55.975428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:55.975398 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb8b132-d7f0-4db8-8fc4-34b424107dd2-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:56.165491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.165387 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:56.166851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.166822 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:56.170147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.170091 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:17:56.256243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.256207 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:17:56.267267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.267218 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8r6rf"] Nov 01 22:17:56.442203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.442123 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" path="/var/lib/kubelet/pods/12da8b2d-cbec-446a-957a-2bd7eae29773/volumes" Nov 01 22:17:56.444796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.444762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" event={"ID":"6fb8b132-d7f0-4db8-8fc4-34b424107dd2","Type":"ContainerDied","Data":"14bd745e8495f76e53a582f4cf7f390a8201717d905b6e31f7d46cd9e78d831f"} Nov 01 22:17:56.444935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.444826 5036 scope.go:122] "RemoveContainer" containerID="2e0ef8af766fba809f39684f635afbba27b6c462dc661126366a9034fc510156" Nov 01 22:17:56.444935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.444928 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d" Nov 01 22:17:56.450223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.450196 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" Nov 01 22:17:56.450351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.450327 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6" event={"ID":"02428bf4-f476-4f4d-9847-5668c194869f","Type":"ContainerDied","Data":"f3990aa2368b75cfc37cf0cde14d0fc80f73b0774b485ee4c5370dcae7df3487"} Nov 01 22:17:56.457095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.456998 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" event={"ID":"52e28a5c-aaaa-4698-9c75-62d149c5c19b","Type":"ContainerDied","Data":"bf2e7730236f7e8afcf96aa590ab2513eb48e203067524cb2b9f189c9784b853"} Nov 01 22:17:56.457429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.457406 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2" Nov 01 22:17:56.462840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.462823 5036 scope.go:122] "RemoveContainer" containerID="c5db5fd3d10009344d93802d152fc6d0f860adc6d03c2ad4dade50fdf22bfdcf" Nov 01 22:17:56.487953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.487927 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:17:56.494980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.494951 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-ckft2"] Nov 01 22:17:56.503614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.503593 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:17:56.509602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.509579 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-4r72d"] Nov 01 22:17:56.520747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.520723 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:17:56.527166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.527131 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-pt6z6"] Nov 01 22:17:56.620269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:56.620237 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:57.135054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:57.134993 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:57.278644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:57.278609 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:57.640471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:57.640401 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:58.461166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:58.458952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:58.479938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:58.479903 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02428bf4-f476-4f4d-9847-5668c194869f" path="/var/lib/kubelet/pods/02428bf4-f476-4f4d-9847-5668c194869f/volumes" Nov 01 22:17:58.481252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:58.481225 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" path="/var/lib/kubelet/pods/52e28a5c-aaaa-4698-9c75-62d149c5c19b/volumes" Nov 01 22:17:58.481943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:58.481921 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" path="/var/lib/kubelet/pods/6fb8b132-d7f0-4db8-8fc4-34b424107dd2/volumes" Nov 01 22:17:58.755997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:58.752704 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:59.629726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:59.629684 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:17:59.934020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:17:59.933910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:00.252029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:00.251941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:00.524297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:00.515487 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:01.453017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:01.445108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:02.299808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:02.299766 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:03.144296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:03.144258 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:03.874886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:03.874845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:04.042124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:04.042091 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v"] Nov 01 22:18:04.046128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:04.046104 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480e49cc-qz74v"] Nov 01 22:18:04.092200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:04.092162 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:04.450409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:04.450371 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e16127-d4e9-4e13-a199-5d342357bd39" path="/var/lib/kubelet/pods/43e16127-d4e9-4e13-a199-5d342357bd39/volumes" Nov 01 22:18:04.741697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:04.741592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:05.538118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:05.538082 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:06.155161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:06.155123 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:06.842385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:06.842341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:07.490226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.490175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:07.828793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.828755 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:18:07.830183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830149 5036 cpu_manager.go:414] "containerMap: removing container" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" Nov 01 22:18:07.830183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830182 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830213 5036 cpu_manager.go:414] "containerMap: removing container" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830222 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830265 5036 cpu_manager.go:414] "containerMap: removing container" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830273 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830308 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" Nov 01 22:18:07.830378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830316 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" Nov 01 22:18:07.830760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830514 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="02428bf4-f476-4f4d-9847-5668c194869f" containerName="istio-proxy" Nov 01 22:18:07.830760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830541 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb8b132-d7f0-4db8-8fc4-34b424107dd2" containerName="istio-proxy" Nov 01 22:18:07.830760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830559 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e28a5c-aaaa-4698-9c75-62d149c5c19b" containerName="istio-proxy" Nov 01 22:18:07.830760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.830575 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="12da8b2d-cbec-446a-957a-2bd7eae29773" containerName="istio-proxy" Nov 01 22:18:07.840478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.840435 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:18:07.840731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.840576 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.849696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.849669 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"openshift-service-ca.crt\"" Nov 01 22:18:07.850099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.849769 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"default-dockercfg-4mgc5\"" Nov 01 22:18:07.850099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.849853 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"kube-root-ca.crt\"" Nov 01 22:18:07.850099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.849927 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"auth-test-model-e3890e-kserve-self-signed-certs\"" Nov 01 22:18:07.880129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.880096 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:07.957452 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957407 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.957655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957486 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.957655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957625 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.957791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957725 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.957934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:07.958012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:07.957949 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56t8\" (UniqueName: \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.063364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.062811 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.063689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.063665 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.064282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.064260 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.064965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.064943 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.065626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.065592 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.065798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.065779 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56t8\" (UniqueName: \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.066046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.063324 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.092521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.066414 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.093148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.064099 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.093340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.070126 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.116152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.083697 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56t8\" (UniqueName: \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.116152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.093055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\") pod \"auth-test-model-e3890e-kserve-548d4b698f-7wwp7\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.175138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.175049 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:08.482274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.482226 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:18:08.628897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.628748 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:08.677469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.676570 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" event={"ID":"e1512f28-2622-4b05-bcaa-1c13a14254da","Type":"ContainerStarted","Data":"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551"} Nov 01 22:18:08.677469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.676629 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" event={"ID":"e1512f28-2622-4b05-bcaa-1c13a14254da","Type":"ContainerStarted","Data":"882b00dd52086e69a07eba13fe0d295ea58ba4729da4de6d023bee552302f0dd"} Nov 01 22:18:08.712465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:08.711988 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" podStartSLOduration=1.7119664399999999 podStartE2EDuration="1.71196644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:08.69992332 +0000 UTC m=+2042.392903817" watchObservedRunningTime="2026-11-01 22:18:08.71196644 +0000 UTC m=+2042.404946926" Nov 01 22:18:09.175199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:09.175165 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:09.338163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:09.338122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:10.126255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:10.126202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:10.854612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:10.854562 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:11.648194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:11.648160 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:12.427133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:12.427097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:13.292397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:13.292355 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:13.949919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:13.949880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:14.551932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.551899 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:18:14.558487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.558424 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.565766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.565736 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:18:14.567741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.567714 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-521734-serving-cert\"" Nov 01 22:18:14.630016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.629913 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:14.686565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.686521 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5nl\" (UniqueName: \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.686941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.686911 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.687213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.687193 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.687711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.687687 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.762947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.762899 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:14.789303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.789276 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.789737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.789708 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5nl\" (UniqueName: \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.789939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.789919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.790037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.789991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.790106 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:18:14.790073 5036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-serving-cert: secret "maas-api-e2e-shared-a-w3-521734-serving-cert" not found Nov 01 22:18:14.790162 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:18:14.790150 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls podName:7b74dc93-39fa-4e73-97d5-3996c7da3273 nodeName:}" failed. No retries permitted until 2026-11-01 22:18:15.290130877 +0000 UTC m=+2048.983111355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls") pod "maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" (UID: "7b74dc93-39fa-4e73-97d5-3996c7da3273") : secret "maas-api-e2e-shared-a-w3-521734-serving-cert" not found Nov 01 22:18:14.790802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.790778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.792916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.792894 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.804616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.804553 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5nl\" (UniqueName: \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:14.960723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:14.960679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:15.040088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.039421 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t"] Nov 01 22:18:15.045411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.045377 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:18:15.051321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.051297 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8b0dbc-6tl8t"] Nov 01 22:18:15.051488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.051390 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.059513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.058967 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:18:15.241469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.230266 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqxw\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.241813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.241780 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.242000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.241981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.242701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.242670 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.242872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.242852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.243065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.243046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.243258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.243239 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.243482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.243464 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.243622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.243605 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.348643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.348545 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.349037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.349012 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.349532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.349504 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.349811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.349792 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.349949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.349932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.350100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350081 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:15.350309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350292 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqxw\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.350424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350410 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.350637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.350758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.350938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.350919 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.351086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.351068 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.353297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.352394 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.353297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.352893 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.353297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.353230 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.353644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.353416 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.354892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.354869 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.355788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.355762 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:15.367373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.367349 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.367710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.367687 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqxw\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.408950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.408914 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:15.498029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.497778 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:15.538955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.538832 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:15.731956 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:15.731914 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b3df78_41a8_4e31_a553_7531916957f1.slice/crio-0e1406a7548890c8306b42efce91f1d52051a000108cbbda805c289860dc337d WatchSource:0}: Error finding container 0e1406a7548890c8306b42efce91f1d52051a000108cbbda805c289860dc337d: Status 404 returned error can't find the container with id 0e1406a7548890c8306b42efce91f1d52051a000108cbbda805c289860dc337d Nov 01 22:18:15.738493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.735963 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:15.738493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.736048 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:15.738493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.736091 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:15.742999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.742970 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:18:15.769977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.769938 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" event={"ID":"d9b3df78-41a8-4e31-a553-7531916957f1","Type":"ContainerStarted","Data":"0e1406a7548890c8306b42efce91f1d52051a000108cbbda805c289860dc337d"} Nov 01 22:18:15.928315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.923290 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:18:15.928315 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:15.925948 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74dc93_39fa_4e73_97d5_3996c7da3273.slice/crio-59e4f33c134aab2e72de9c944224486020569221ada56dc7f93aea7fe8fdaa9d WatchSource:0}: Error finding container 59e4f33c134aab2e72de9c944224486020569221ada56dc7f93aea7fe8fdaa9d: Status 404 returned error can't find the container with id 59e4f33c134aab2e72de9c944224486020569221ada56dc7f93aea7fe8fdaa9d Nov 01 22:18:15.929979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:15.929953 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:18:16.441493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.441460 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7" path="/var/lib/kubelet/pods/b8563d3c-b9b0-4946-ac7f-8abb2e26f1a7/volumes" Nov 01 22:18:16.443423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.443398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:16.812603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.808318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" event={"ID":"7b74dc93-39fa-4e73-97d5-3996c7da3273","Type":"ContainerStarted","Data":"7320cf6acc35d17dab50635b891bce18237ea2a80f1fad1efab9b133f79c2f23"} Nov 01 22:18:16.812603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.808389 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" event={"ID":"7b74dc93-39fa-4e73-97d5-3996c7da3273","Type":"ContainerStarted","Data":"59e4f33c134aab2e72de9c944224486020569221ada56dc7f93aea7fe8fdaa9d"} Nov 01 22:18:16.812603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.809147 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:16.818730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.818694 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" event={"ID":"d9b3df78-41a8-4e31-a553-7531916957f1","Type":"ContainerStarted","Data":"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218"} Nov 01 22:18:16.837839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.837127 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" podStartSLOduration=2.437018638 podStartE2EDuration="2.83711185s" totalImagesPullingTime="400.093212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:16.834910274 +0000 UTC m=+2050.527890760" watchObservedRunningTime="2026-11-01 22:18:16.83711185 +0000 UTC m=+2050.530092338" Nov 01 22:18:16.865042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:16.864963 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podStartSLOduration=1.864950673 podStartE2EDuration="1.864950673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:16.861684706 +0000 UTC m=+2050.554665194" watchObservedRunningTime="2026-11-01 22:18:16.864950673 +0000 UTC m=+2050.557931158" Nov 01 22:18:17.410501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:17.409880 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:17.414113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:17.411799 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:17.414113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:17.411860 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:17.532905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:17.532859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:18.175759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.175720 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:18.221919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.221884 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:18.410019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.409977 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:18.410230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.410032 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:18.415532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.415318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:18.857854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:18.857823 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:18:18.868291 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:18:18.868261 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:18:19.234106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:19.234018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:19.409843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:19.409803 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:19.410121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:19.410092 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:20.236037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:20.236006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:20.412907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:20.412866 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:20.413201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:20.413168 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:21.353115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:21.353046 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:21.412463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:21.411907 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:21.412463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:21.411987 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:22.241008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:22.240964 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:22.409315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:22.409273 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 01 22:18:22.409857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:22.409343 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 01 22:18:23.103107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.103054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:23.416434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.416348 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:23.417673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.417642 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:23.419842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.419819 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:18:23.781396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.781363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:23.847066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:23.847028 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:18:24.071650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:24.071574 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:18:24.668080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:24.668045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:25.661847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:25.661809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:26.643132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:26.643093 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:27.513189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:27.513140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:28.284818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:28.284774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:28.948424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:28.948381 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:29.782961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:29.782917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:30.072509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.072409 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:18:30.082867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.082838 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.090641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.090582 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:18:30.092574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.092531 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"openshift-service-ca.crt\"" Nov 01 22:18:30.092733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.092578 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"auth-test-model-51e3b7-kserve-self-signed-certs\"" Nov 01 22:18:30.092733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.092644 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"default-dockercfg-wcw8g\"" Nov 01 22:18:30.092733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.092715 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"kube-root-ca.crt\"" Nov 01 22:18:30.134084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.134048 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:30.170762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.170734 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:18:30.176233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.176210 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.327989 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328068 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328105 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328143 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328205 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328241 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328298 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328369 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328566 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dkj\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328631 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328697 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328738 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328810 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.329521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.328885 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnvj\" (UniqueName: \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440303 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440378 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440487 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440548 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440660 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440701 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dkj\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440763 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440873 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440934 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.440998 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnvj\" (UniqueName: \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.441052 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.441108 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.441143 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.443075 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.444465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.443686 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.460229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dkj\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.463730 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.466222 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.466913 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.467320 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.467596 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.468196 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.469160 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.469434 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.473065 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.480636 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.485509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.485081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.507768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.507737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnvj\" (UniqueName: \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\") pod \"auth-test-model-51e3b7-kserve-765c7d57df-v8h6l\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.526705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.526676 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:18:30.621103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.621022 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:30.732871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.716502 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:30.732871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:30.723265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:31.114827 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:31.114791 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c537dcc_54f4_4512_878d_39dac142d205.slice/crio-b89b79e8315826e9752c05d38d1780ce2c421d1235b2af7241b3226591798d1c WatchSource:0}: Error finding container b89b79e8315826e9752c05d38d1780ce2c421d1235b2af7241b3226591798d1c: Status 404 returned error can't find the container with id b89b79e8315826e9752c05d38d1780ce2c421d1235b2af7241b3226591798d1c Nov 01 22:18:31.116008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.115978 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:18:31.134812 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:31.134755 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5178aae_9f8e_43db_9c99_56aae2a3aff5.slice/crio-3e17b34950292fff0f7165f42be9d8a0b6dfbb6413a417032a61e09912e2e09e WatchSource:0}: Error finding container 3e17b34950292fff0f7165f42be9d8a0b6dfbb6413a417032a61e09912e2e09e: Status 404 returned error can't find the container with id 3e17b34950292fff0f7165f42be9d8a0b6dfbb6413a417032a61e09912e2e09e Nov 01 22:18:31.135837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.135816 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:18:31.138098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.138069 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:31.138167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.138137 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:31.138227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.138171 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:31.417694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:31.417598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:32.075524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.075482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" event={"ID":"d5178aae-9f8e-43db-9c99-56aae2a3aff5","Type":"ContainerStarted","Data":"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794"} Nov 01 22:18:32.075740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.075552 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" event={"ID":"d5178aae-9f8e-43db-9c99-56aae2a3aff5","Type":"ContainerStarted","Data":"3e17b34950292fff0f7165f42be9d8a0b6dfbb6413a417032a61e09912e2e09e"} Nov 01 22:18:32.080498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.077648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" event={"ID":"9c537dcc-54f4-4512-878d-39dac142d205","Type":"ContainerStarted","Data":"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01"} Nov 01 22:18:32.080498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.077694 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" event={"ID":"9c537dcc-54f4-4512-878d-39dac142d205","Type":"ContainerStarted","Data":"b89b79e8315826e9752c05d38d1780ce2c421d1235b2af7241b3226591798d1c"} Nov 01 22:18:32.111883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.111820 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podStartSLOduration=2.111798919 podStartE2EDuration="2.111798919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:32.103722011 +0000 UTC m=+2065.796702499" watchObservedRunningTime="2026-11-01 22:18:32.111798919 +0000 UTC m=+2065.804779411" Nov 01 22:18:32.128384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.128331 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" podStartSLOduration=2.128317693 podStartE2EDuration="2.128317693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:32.12408447 +0000 UTC m=+2065.817064961" watchObservedRunningTime="2026-11-01 22:18:32.128317693 +0000 UTC m=+2065.821298180" Nov 01 22:18:32.351725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.351632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:32.621551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.621429 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:32.623369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.623333 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:32.623578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.623395 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:32.718121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:32.718081 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:33.137296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:33.137260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:33.622414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:33.622377 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:33.622633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:33.622439 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:33.970800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:33.970701 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:34.621961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:34.621923 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:34.622398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:34.622370 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:34.772985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:34.772956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:35.627467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:35.625649 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:35.627467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:35.625709 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:35.946049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:35.945950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:36.623858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:36.623700 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:36.623858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:36.623757 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:36.816337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:36.816292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:37.544012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:37.543977 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:37.627901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:37.627857 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 01 22:18:37.628129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:37.627919 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 01 22:18:37.674166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:37.674114 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:37.996179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:37.996075 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:38.282033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.281991 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:38.412973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.412927 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:38.591092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.591002 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:38.765653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.763679 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:38.765653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.765494 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:38.796742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:38.796711 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:18:39.245828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:39.245794 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:40.025355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:40.025312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:40.718529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:40.718495 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:40.731544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:40.731509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:40.985387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:40.985289 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:41.228908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:41.228873 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:18:41.733970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:41.733926 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:41.978330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:41.978292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:42.107187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:42.107153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:42.584846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:42.584806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:43.161065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:43.161028 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:43.316166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:43.316125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:43.683804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:43.683765 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:44.515612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:44.515574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:45.425352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:45.425298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:46.146185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:46.146139 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:46.837073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:46.837019 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:47.256046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.255962 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:18:47.279306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.279273 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.307864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.307831 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:18:47.313896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.311679 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-de0cba-openshift-default-dockercfg-rkxw4\"" Nov 01 22:18:47.314080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.313721 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.314240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.314224 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.314435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.314421 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.314711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.314691 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.314833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.314819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.314931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.314919 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.315048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.315032 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsw7w\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.315279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.315256 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.315373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.315317 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.416893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.416846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417038 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417080 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417109 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417157 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsw7w\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417224 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417283 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417344 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417416 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417481 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417548 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417804 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.417942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.417920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.418523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.418496 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.419568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.419544 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.420332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.420309 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.429563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.429540 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.429777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.429756 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsw7w\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.624666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.624629 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:47.765499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.740872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:47.942139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.942110 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:18:47.944631 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:47.944603 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2476a2b_8eb0_4b13_ae22_47c72902a67d.slice/crio-430dbdc11da43fce97afd70c6278d72f933fbe608dc89d448e63603bf4a17304 WatchSource:0}: Error finding container 430dbdc11da43fce97afd70c6278d72f933fbe608dc89d448e63603bf4a17304: Status 404 returned error can't find the container with id 430dbdc11da43fce97afd70c6278d72f933fbe608dc89d448e63603bf4a17304 Nov 01 22:18:47.948132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.948089 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:47.948229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.948182 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:47.948279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:47.948226 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:18:48.301748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:48.301702 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" event={"ID":"e2476a2b-8eb0-4b13-ae22-47c72902a67d","Type":"ContainerStarted","Data":"53744ef78a182040ea3d717173c14dde2a5b4971a52b8db418d47a529ddba18b"} Nov 01 22:18:48.302166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:48.301783 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" event={"ID":"e2476a2b-8eb0-4b13-ae22-47c72902a67d","Type":"ContainerStarted","Data":"430dbdc11da43fce97afd70c6278d72f933fbe608dc89d448e63603bf4a17304"} Nov 01 22:18:48.330261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:48.330211 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podStartSLOduration=1.3301988200000001 podStartE2EDuration="1.33019882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:48.325030535 +0000 UTC m=+2082.018011057" watchObservedRunningTime="2026-11-01 22:18:48.33019882 +0000 UTC m=+2082.023179306" Nov 01 22:18:48.504496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:48.504430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:48.626340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:48.626252 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:48.866561 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:18:48.866511 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:18:49.313546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:49.313512 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:49.630648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:49.630550 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:49.630925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:49.630886 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:50.012435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:50.012342 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:50.628626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:50.628586 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:50.629297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:50.629256 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:50.875154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:50.875115 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:51.625049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:51.625014 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:51.625359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:51.625327 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:51.642050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:51.642009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:52.215854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.215818 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv"] Nov 01 22:18:52.224492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.224460 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.231058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.229993 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:18:52.231058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.230936 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv"] Nov 01 22:18:52.231400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.231372 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:18:52.311774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.311722 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.312220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.312195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.312569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.312514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwz5\" (UniqueName: \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.416061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.414320 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.416061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.414508 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.416061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.414632 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwz5\" (UniqueName: \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.416061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.415847 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.417806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.417765 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.427998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.427941 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwz5\" (UniqueName: \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\") pod \"maas-api-revoke-sub-b91ac00c-9mnrv\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.516578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.507088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:52.555249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.553913 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:52.625916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.625769 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:52.625916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.625823 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:52.915857 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:18:52.915827 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18a5f2b_3f20_4779_acf8_4ac85aaea68a.slice/crio-5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250 WatchSource:0}: Error finding container 5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250: Status 404 returned error can't find the container with id 5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250 Nov 01 22:18:52.919310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:52.919283 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv"] Nov 01 22:18:53.227592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.227458 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:53.359358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.359318 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" event={"ID":"f18a5f2b-3f20-4779-acf8-4ac85aaea68a","Type":"ContainerStarted","Data":"5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250"} Nov 01 22:18:53.647657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.647569 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:53.647657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.647641 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:53.652389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.652325 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:18:53.662926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:53.662893 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:54.140752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.140711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:54.371666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.371578 5036 generic.go:356] "Generic (PLEG): container finished" podID="f18a5f2b-3f20-4779-acf8-4ac85aaea68a" containerID="a143ed25abc126efd440c79be62adef3d26849b840e5b20e075c758430e5e739" exitCode=0 Nov 01 22:18:54.371666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.371633 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" event={"ID":"f18a5f2b-3f20-4779-acf8-4ac85aaea68a","Type":"ContainerDied","Data":"a143ed25abc126efd440c79be62adef3d26849b840e5b20e075c758430e5e739"} Nov 01 22:18:54.625434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.625355 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Nov 01 22:18:54.625891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.625852 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Nov 01 22:18:54.825325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:54.825261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:55.543396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.543361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:55.633513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.631316 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:55.637519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.634615 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:55.637519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.635651 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:55.637519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.636326 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:18:55.796079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.796045 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\") pod \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " Nov 01 22:18:55.796282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.796237 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\" (UniqueName: \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\") pod \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " Nov 01 22:18:55.796354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.796283 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\") pod \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\" (UID: \"f18a5f2b-3f20-4779-acf8-4ac85aaea68a\") " Nov 01 22:18:55.796474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.796432 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca" pod "f18a5f2b-3f20-4779-acf8-4ac85aaea68a" (UID: "f18a5f2b-3f20-4779-acf8-4ac85aaea68a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:18:55.797372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.797331 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:18:55.798995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.798968 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5" pod "f18a5f2b-3f20-4779-acf8-4ac85aaea68a" (UID: "f18a5f2b-3f20-4779-acf8-4ac85aaea68a"). InnerVolumeSpecName "kube-api-access-zgwz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:18:55.800182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.800162 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca" pod "f18a5f2b-3f20-4779-acf8-4ac85aaea68a" (UID: "f18a5f2b-3f20-4779-acf8-4ac85aaea68a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:18:55.899895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.899798 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgwz5\" (UniqueName: \"kubernetes.io/projected/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-kube-api-access-zgwz5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:18:55.899895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:55.899839 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18a5f2b-3f20-4779-acf8-4ac85aaea68a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:18:56.326732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:56.326694 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:56.394009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:56.393965 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" Nov 01 22:18:56.394009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:56.393990 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv" event={"ID":"f18a5f2b-3f20-4779-acf8-4ac85aaea68a","Type":"ContainerDied","Data":"5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250"} Nov 01 22:18:56.394293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:56.394052 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1bc49aa9dc0c15106f3474cb96433deaa203a8401c67948744064f3ac53250" Nov 01 22:18:57.067990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:57.067955 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:57.822861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:57.822824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:18:59.918000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:18:59.917955 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:00.056848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:00.056812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:00.284664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:00.284612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:00.850041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:00.850006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:02.117023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:02.116983 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:02.830200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:02.830154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:03.563020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:03.562985 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:04.377691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:04.377636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:05.088707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:05.088671 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:05.818510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:05.818475 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:06.625576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:06.625540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:06.807939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:06.807887 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:19:06.860929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:06.860889 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:07.577469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.576640 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:07.750940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.750915 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:19:07.895103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.895018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:07.945101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.945045 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh"] Nov 01 22:19:07.946234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.946200 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f18a5f2b-3f20-4779-acf8-4ac85aaea68a" containerName="revoke-keys" Nov 01 22:19:07.946459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.946427 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18a5f2b-3f20-4779-acf8-4ac85aaea68a" containerName="revoke-keys" Nov 01 22:19:07.946821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.946804 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18a5f2b-3f20-4779-acf8-4ac85aaea68a" containerName="revoke-keys" Nov 01 22:19:07.956544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.956516 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:07.961568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.961542 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:19:07.964595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.961973 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh"] Nov 01 22:19:07.964769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:07.962668 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:19:08.018737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.018699 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.018918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.018743 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbsw\" (UniqueName: \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.019052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.019036 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.120768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.120730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.120950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.120828 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.120950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.120862 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbsw\" (UniqueName: \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.121779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.121748 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.123345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.123321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.133493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.133466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbsw\" (UniqueName: \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\") pod \"maas-api-revoke-sub-c8a249fd-v8mxh\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.302347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.302314 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:08.521427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.521311 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:19:08.545925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.545848 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:08.645533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.639525 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh"] Nov 01 22:19:08.645533 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:19:08.641572 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172054ca_402d_4bf7_af9e_132dbacede8e.slice/crio-256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021 WatchSource:0}: Error finding container 256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021: Status 404 returned error can't find the container with id 256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021 Nov 01 22:19:08.672987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.672939 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" event={"ID":"172054ca-402d-4bf7-af9e-132dbacede8e","Type":"ContainerStarted","Data":"256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021"} Nov 01 22:19:08.974915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:08.974875 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" podUID="e1512f28-2622-4b05-bcaa-1c13a14254da" containerName="main" containerID="cri-o://50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551" gracePeriod=45 Nov 01 22:19:09.301275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.301250 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:19:09.338687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.338650 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.338909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.338703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.338909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.338819 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.338909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.338901 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\" (UniqueName: \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.339094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.338949 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.339094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.339044 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\") pod \"e1512f28-2622-4b05-bcaa-1c13a14254da\" (UID: \"e1512f28-2622-4b05-bcaa-1c13a14254da\") " Nov 01 22:19:09.340736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.340700 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:09.340879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.340790 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:09.341223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.341068 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:09.342514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.342421 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:09.342743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.342722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:09.350893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.350865 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8" pod "e1512f28-2622-4b05-bcaa-1c13a14254da" (UID: "e1512f28-2622-4b05-bcaa-1c13a14254da"). InnerVolumeSpecName "kube-api-access-p56t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:09.425218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.425137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450285 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450328 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450343 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450358 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56t8\" (UniqueName: \"kubernetes.io/projected/e1512f28-2622-4b05-bcaa-1c13a14254da-kube-api-access-p56t8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450373 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1512f28-2622-4b05-bcaa-1c13a14254da-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.454265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.450389 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1512f28-2622-4b05-bcaa-1c13a14254da-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:09.685749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.685658 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" event={"ID":"172054ca-402d-4bf7-af9e-132dbacede8e","Type":"ContainerStarted","Data":"2d382f15d4ae9bf7d089643b45a3c0488b204d3e435450466137f1c2e9a62083"} Nov 01 22:19:09.689720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.689685 5036 generic.go:356] "Generic (PLEG): container finished" podID="e1512f28-2622-4b05-bcaa-1c13a14254da" containerID="50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551" exitCode=0 Nov 01 22:19:09.689872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.689803 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" event={"ID":"e1512f28-2622-4b05-bcaa-1c13a14254da","Type":"ContainerDied","Data":"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551"} Nov 01 22:19:09.689872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.689835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" event={"ID":"e1512f28-2622-4b05-bcaa-1c13a14254da","Type":"ContainerDied","Data":"882b00dd52086e69a07eba13fe0d295ea58ba4729da4de6d023bee552302f0dd"} Nov 01 22:19:09.689872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.689855 5036 scope.go:122] "RemoveContainer" containerID="50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551" Nov 01 22:19:09.690148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.690121 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7" Nov 01 22:19:09.706952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.706931 5036 scope.go:122] "RemoveContainer" containerID="50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551" Nov 01 22:19:09.707058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.707013 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" podStartSLOduration=2.707000845 podStartE2EDuration="2.707000845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:09.706641423 +0000 UTC m=+2103.399621910" watchObservedRunningTime="2026-11-01 22:19:09.707000845 +0000 UTC m=+2103.399981330" Nov 01 22:19:09.707236 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:19:09.707213 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551\": container with ID starting with 50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551 not found: ID does not exist" containerID="50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551" Nov 01 22:19:09.707309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.707246 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551"} err="failed to get container status \"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551\": rpc error: code = NotFound desc = could not find container \"50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551\": container with ID starting with 50585fea25167da975308de00a02e154ba8b6a0b5a0e8cbf79fdfe5638ea2551 not found: ID does not exist" Nov 01 22:19:09.727937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.727911 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:19:09.732563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:09.732539 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/auth-test-model-e3890e-kserve-548d4b698f-7wwp7"] Nov 01 22:19:10.142387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:10.142349 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:10.440487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:10.440373 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1512f28-2622-4b05-bcaa-1c13a14254da" path="/var/lib/kubelet/pods/e1512f28-2622-4b05-bcaa-1c13a14254da/volumes" Nov 01 22:19:10.700282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:10.700195 5036 generic.go:356] "Generic (PLEG): container finished" podID="172054ca-402d-4bf7-af9e-132dbacede8e" containerID="2d382f15d4ae9bf7d089643b45a3c0488b204d3e435450466137f1c2e9a62083" exitCode=0 Nov 01 22:19:10.700481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:10.700274 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" event={"ID":"172054ca-402d-4bf7-af9e-132dbacede8e","Type":"ContainerDied","Data":"2d382f15d4ae9bf7d089643b45a3c0488b204d3e435450466137f1c2e9a62083"} Nov 01 22:19:10.939196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:10.939152 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:11.688467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.682997 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:11.890485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.890408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:11.922595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.922567 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:11.994707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.994617 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\" (UniqueName: \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\") pod \"172054ca-402d-4bf7-af9e-132dbacede8e\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " Nov 01 22:19:11.994868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.994733 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\") pod \"172054ca-402d-4bf7-af9e-132dbacede8e\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " Nov 01 22:19:11.994868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.994858 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\") pod \"172054ca-402d-4bf7-af9e-132dbacede8e\" (UID: \"172054ca-402d-4bf7-af9e-132dbacede8e\") " Nov 01 22:19:11.996043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.996013 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca" pod "172054ca-402d-4bf7-af9e-132dbacede8e" (UID: "172054ca-402d-4bf7-af9e-132dbacede8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:11.998822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:11.998796 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca" pod "172054ca-402d-4bf7-af9e-132dbacede8e" (UID: "172054ca-402d-4bf7-af9e-132dbacede8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:12.000266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.000212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw" pod "172054ca-402d-4bf7-af9e-132dbacede8e" (UID: "172054ca-402d-4bf7-af9e-132dbacede8e"). InnerVolumeSpecName "kube-api-access-gpbsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:12.097806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.097763 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbsw\" (UniqueName: \"kubernetes.io/projected/172054ca-402d-4bf7-af9e-132dbacede8e-kube-api-access-gpbsw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:12.097806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.097805 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:12.097998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.097821 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172054ca-402d-4bf7-af9e-132dbacede8e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:12.544172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.544129 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:12.734938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.734856 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" Nov 01 22:19:12.734938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.734879 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh" event={"ID":"172054ca-402d-4bf7-af9e-132dbacede8e","Type":"ContainerDied","Data":"256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021"} Nov 01 22:19:12.735405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:12.734938 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256093ec210a91982559144ecc0070e057096ada10c238a883f31f69a8ca8021" Nov 01 22:19:13.327649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:13.326056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:14.049344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:14.049300 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:14.736507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:14.736467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:15.436513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:15.436479 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:16.222303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:16.218291 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:16.706809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:16.706770 5036 scope.go:122] "RemoveContainer" containerID="9d3c9c47aae6ced0f52be758c96bb4708b9f89aa6ac9afd15f15a55d91aca87f" Nov 01 22:19:16.720241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:16.720213 5036 scope.go:122] "RemoveContainer" containerID="db8401790f855e1c421ac7be6ec7070e02408e088284448ddcbdadf60f810693" Nov 01 22:19:16.949516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:16.949478 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:17.737154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:17.737049 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:18.640866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:18.640828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:18.865600 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:19:18.865557 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:19:19.314860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:19.314822 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:20.186150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.186093 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:20.891131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.891032 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:20.908672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.908641 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:19:20.909977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.909949 5036 cpu_manager.go:414] "containerMap: removing container" podUID="172054ca-402d-4bf7-af9e-132dbacede8e" containerName="revoke-keys" Nov 01 22:19:20.910219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.910196 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172054ca-402d-4bf7-af9e-132dbacede8e" containerName="revoke-keys" Nov 01 22:19:20.910325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.910242 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e1512f28-2622-4b05-bcaa-1c13a14254da" containerName="main" Nov 01 22:19:20.910325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.910251 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1512f28-2622-4b05-bcaa-1c13a14254da" containerName="main" Nov 01 22:19:20.910619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.910489 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="172054ca-402d-4bf7-af9e-132dbacede8e" containerName="revoke-keys" Nov 01 22:19:20.910619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.910513 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1512f28-2622-4b05-bcaa-1c13a14254da" containerName="main" Nov 01 22:19:20.928491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.916929 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.929346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.929321 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:19:20.933665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.933644 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"kube-root-ca.crt\"" Nov 01 22:19:20.935658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.934831 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"openshift-service-ca.crt\"" Nov 01 22:19:20.935658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.935011 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"default-dockercfg-4mgc5\"" Nov 01 22:19:20.935658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.935087 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e3890e\"/\"sub-test-model-e3890e-kserve-self-signed-certs\"" Nov 01 22:19:20.957043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957016 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.957192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957121 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.957192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.957364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957267 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.957364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957305 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2b7\" (UniqueName: \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:20.957364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:20.957339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059472 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059619 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059662 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2b7\" (UniqueName: \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059699 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059858 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.059947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.059902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.060327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.060302 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.060534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.060508 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.062153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.062126 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.062261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.062190 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.088512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.088485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2b7\" (UniqueName: \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\") pod \"sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.107002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.106970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:21.252225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.250760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:21.700756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.700725 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:19:21.761956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.761921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:21.855004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.851623 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" event={"ID":"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f","Type":"ContainerStarted","Data":"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd"} Nov 01 22:19:21.855004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.851716 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" event={"ID":"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f","Type":"ContainerStarted","Data":"73cf32b7eff7c135a313b616927722a45e491bd6165878c313022d84d0c9b661"} Nov 01 22:19:21.915141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:21.908634 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" podUID="9c537dcc-54f4-4512-878d-39dac142d205" containerName="main" containerID="cri-o://32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01" gracePeriod=45 Nov 01 22:19:22.014132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.013936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:22.226421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.226365 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" podUID="9c537dcc-54f4-4512-878d-39dac142d205" containerName="main" probeResult="failure" output="Get \"https://10.132.1.33:8000/ready\": dial tcp 10.132.1.33:8000: connect: connection refused" Nov 01 22:19:22.257192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.257125 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" podStartSLOduration=2.257102386 podStartE2EDuration="2.257102386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:21.882237535 +0000 UTC m=+2115.575218032" watchObservedRunningTime="2026-11-01 22:19:22.257102386 +0000 UTC m=+2115.950082872" Nov 01 22:19:22.264574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.262593 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:22.267394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.267366 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v"] Nov 01 22:19:22.278982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.278330 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.282088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.282065 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v"] Nov 01 22:19:22.283301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.283281 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:19:22.283638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.283619 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:19:22.435753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.434809 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7c7\" (UniqueName: \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.435753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.435029 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.435753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.435100 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.537404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.537366 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.537549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.537470 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.537787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.537768 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7c7\" (UniqueName: \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.538488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.538460 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.541851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.541826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.553074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.553046 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7c7\" (UniqueName: \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\") pod \"maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.626327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.622488 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:22.661281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.660686 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:22.666173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.665891 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.740605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.740742 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.740788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.740842 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\" (UniqueName: \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.740931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.741343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.741075 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\") pod \"9c537dcc-54f4-4512-878d-39dac142d205\" (UID: \"9c537dcc-54f4-4512-878d-39dac142d205\") " Nov 01 22:19:22.742521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.742325 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:22.743003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.742698 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:22.743485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.743156 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:22.757467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.756760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:22.757722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.757680 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:22.758230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.758198 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj" pod "9c537dcc-54f4-4512-878d-39dac142d205" (UID: "9c537dcc-54f4-4512-878d-39dac142d205"). InnerVolumeSpecName "kube-api-access-2qnvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844432 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844487 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c537dcc-54f4-4512-878d-39dac142d205-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844504 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844521 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnvj\" (UniqueName: \"kubernetes.io/projected/9c537dcc-54f4-4512-878d-39dac142d205-kube-api-access-2qnvj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844535 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.845326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.844548 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c537dcc-54f4-4512-878d-39dac142d205-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:22.867786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.867756 5036 generic.go:356] "Generic (PLEG): container finished" podID="9c537dcc-54f4-4512-878d-39dac142d205" containerID="32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01" exitCode=0 Nov 01 22:19:22.867938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.867805 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" Nov 01 22:19:22.867938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.867835 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" event={"ID":"9c537dcc-54f4-4512-878d-39dac142d205","Type":"ContainerDied","Data":"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01"} Nov 01 22:19:22.867938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.867890 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l" event={"ID":"9c537dcc-54f4-4512-878d-39dac142d205","Type":"ContainerDied","Data":"b89b79e8315826e9752c05d38d1780ce2c421d1235b2af7241b3226591798d1c"} Nov 01 22:19:22.867938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.867913 5036 scope.go:122] "RemoveContainer" containerID="32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01" Nov 01 22:19:22.881843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.881825 5036 scope.go:122] "RemoveContainer" containerID="32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01" Nov 01 22:19:22.882116 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:19:22.882084 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01\": container with ID starting with 32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01 not found: ID does not exist" containerID="32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01" Nov 01 22:19:22.882212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.882118 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01"} err="failed to get container status \"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01\": rpc error: code = NotFound desc = could not find container \"32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01\": container with ID starting with 32177ba05ff7828fd9838a167719e9ffd3b8af983d54205e8e7dce88c7f2bd01 not found: ID does not exist" Nov 01 22:19:22.897764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.897740 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:19:22.906062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.906035 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/auth-test-model-51e3b7-kserve-765c7d57df-v8h6l"] Nov 01 22:19:22.925066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.925040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:22.942497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:22.939930 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v"] Nov 01 22:19:23.663816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:23.663734 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:23.883539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:23.883488 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" event={"ID":"7236adf9-de51-4441-848b-4dd8653477dd","Type":"ContainerStarted","Data":"5201b43369a1e6a2ebb78c2f1182fcbee083ef087d8796637e61553d88d0894d"} Nov 01 22:19:23.883739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:23.883555 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" event={"ID":"7236adf9-de51-4441-848b-4dd8653477dd","Type":"ContainerStarted","Data":"995089a0b71539c64ef2deeed047dae5ff73e326821ca4b5bd4188a9f8c8ff85"} Nov 01 22:19:23.907481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:23.907404 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" podStartSLOduration=1.9073874709999998 podStartE2EDuration="1.907387471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:23.901459526 +0000 UTC m=+2117.594440005" watchObservedRunningTime="2026-11-01 22:19:23.907387471 +0000 UTC m=+2117.600367956" Nov 01 22:19:24.374860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:24.374802 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:24.442375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:24.442296 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c537dcc-54f4-4512-878d-39dac142d205" path="/var/lib/kubelet/pods/9c537dcc-54f4-4512-878d-39dac142d205/volumes" Nov 01 22:19:24.895120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:24.895087 5036 generic.go:356] "Generic (PLEG): container finished" podID="7236adf9-de51-4441-848b-4dd8653477dd" containerID="5201b43369a1e6a2ebb78c2f1182fcbee083ef087d8796637e61553d88d0894d" exitCode=0 Nov 01 22:19:24.895568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:24.895175 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" event={"ID":"7236adf9-de51-4441-848b-4dd8653477dd","Type":"ContainerDied","Data":"5201b43369a1e6a2ebb78c2f1182fcbee083ef087d8796637e61553d88d0894d"} Nov 01 22:19:25.128742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:25.128702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:25.823465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:25.820517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:26.096362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.096297 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:26.211314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.211282 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\" (UniqueName: \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\") pod \"7236adf9-de51-4441-848b-4dd8653477dd\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " Nov 01 22:19:26.211511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.211340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\") pod \"7236adf9-de51-4441-848b-4dd8653477dd\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " Nov 01 22:19:26.211582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.211528 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\") pod \"7236adf9-de51-4441-848b-4dd8653477dd\" (UID: \"7236adf9-de51-4441-848b-4dd8653477dd\") " Nov 01 22:19:26.211798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.211763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca" pod "7236adf9-de51-4441-848b-4dd8653477dd" (UID: "7236adf9-de51-4441-848b-4dd8653477dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:26.212302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.212279 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:26.213643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.213609 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7" pod "7236adf9-de51-4441-848b-4dd8653477dd" (UID: "7236adf9-de51-4441-848b-4dd8653477dd"). InnerVolumeSpecName "kube-api-access-mx7c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:26.213742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.213676 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca" pod "7236adf9-de51-4441-848b-4dd8653477dd" (UID: "7236adf9-de51-4441-848b-4dd8653477dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:26.314324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.314292 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7236adf9-de51-4441-848b-4dd8653477dd-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:26.314488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.314328 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7c7\" (UniqueName: \"kubernetes.io/projected/7236adf9-de51-4441-848b-4dd8653477dd-kube-api-access-mx7c7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:26.649370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.649333 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:26.919485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.919404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" event={"ID":"7236adf9-de51-4441-848b-4dd8653477dd","Type":"ContainerDied","Data":"995089a0b71539c64ef2deeed047dae5ff73e326821ca4b5bd4188a9f8c8ff85"} Nov 01 22:19:26.919667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.919523 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v" Nov 01 22:19:26.919667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:26.919527 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995089a0b71539c64ef2deeed047dae5ff73e326821ca4b5bd4188a9f8c8ff85" Nov 01 22:19:27.525408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:27.525371 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:28.156527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:28.156491 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:28.933423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:28.933379 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:29.730909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:29.730873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:30.430793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.430271 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:30.486227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.485769 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490390 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9c537dcc-54f4-4512-878d-39dac142d205" containerName="main" Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490422 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c537dcc-54f4-4512-878d-39dac142d205" containerName="main" Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490460 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7236adf9-de51-4441-848b-4dd8653477dd" containerName="revoke-keys" Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490469 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7236adf9-de51-4441-848b-4dd8653477dd" containerName="revoke-keys" Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490693 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c537dcc-54f4-4512-878d-39dac142d205" containerName="main" Nov 01 22:19:30.491979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.490721 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7236adf9-de51-4441-848b-4dd8653477dd" containerName="revoke-keys" Nov 01 22:19:30.500923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.500892 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.514699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.514666 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-51e3b7-openshift-default-dockercfg-5ns2r\"" Nov 01 22:19:30.526134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.522493 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.589747 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.589862 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52kw\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590054 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590295 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590407 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590551 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590644 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590688 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.595489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.590760 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.691894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.691799 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.691894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.691871 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.691941 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692001 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692024 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692063 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692173 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692229 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52kw\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692293 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.692671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692646 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.693107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.693107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692922 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.693107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.692977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.693107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.693061 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.694982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.694956 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.695218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.695196 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.704495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.704466 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.704583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.704538 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52kw\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:30.847025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:30.846991 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:31.152193 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:19:31.152161 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c44b2a_a745_4bca_9142_b05b85c4663f.slice/crio-aa6bd5084707e99bac77b3c16c7f4b9c46a34cad055c37dd97fc59503e039212 WatchSource:0}: Error finding container aa6bd5084707e99bac77b3c16c7f4b9c46a34cad055c37dd97fc59503e039212: Status 404 returned error can't find the container with id aa6bd5084707e99bac77b3c16c7f4b9c46a34cad055c37dd97fc59503e039212 Nov 01 22:19:31.153660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.153627 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:19:31.156251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.156207 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:19:31.156351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.156295 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:19:31.156351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.156340 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:19:31.235322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.235293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:31.259526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.258552 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:31.321347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.320089 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:31.950674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:31.950625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:32.049079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.049039 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" event={"ID":"95c44b2a-a745-4bca-9142-b05b85c4663f","Type":"ContainerStarted","Data":"1b5328cb60a9b1c89282da01ff2f0a861af806afd2316ceb3bf88c21cae3afe5"} Nov 01 22:19:32.049372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.049346 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" event={"ID":"95c44b2a-a745-4bca-9142-b05b85c4663f","Type":"ContainerStarted","Data":"aa6bd5084707e99bac77b3c16c7f4b9c46a34cad055c37dd97fc59503e039212"} Nov 01 22:19:32.061195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.061165 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:19:32.074566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.074521 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podStartSLOduration=2.074509487 podStartE2EDuration="2.074509487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:32.072187817 +0000 UTC m=+2125.765168304" watchObservedRunningTime="2026-11-01 22:19:32.074509487 +0000 UTC m=+2125.767489972" Nov 01 22:19:32.466466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.445878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:32.571770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.571737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:32.778467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.778347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:32.856510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.854019 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:32.856510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.854143 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:32.856510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:32.854187 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:33.462540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.462339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:33.742539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.739872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:33.855467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.854552 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:33.855467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.854633 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:33.961470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.960876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:33.961470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.961035 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v"] Nov 01 22:19:33.999512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:33.990705 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dfdb11-mjf2v"] Nov 01 22:19:34.129468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.128159 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:34.442916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.442879 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7236adf9-de51-4441-848b-4dd8653477dd" path="/var/lib/kubelet/pods/7236adf9-de51-4441-848b-4dd8653477dd/volumes" Nov 01 22:19:34.463043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.463012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:34.820622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.820576 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:34.848189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.848014 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:34.848189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:34.848080 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:35.551664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:35.551627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:35.848232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:35.848149 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:35.848232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:35.848220 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:36.226507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.226376 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:36.764561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.764526 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:19:36.775348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.775319 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.778363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.778312 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:19:36.784129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.783538 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-521734\"/\"openshift-service-ca.crt\"" Nov 01 22:19:36.784858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.784653 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-521734\"/\"kube-root-ca.crt\"" Nov 01 22:19:36.784858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.784753 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-521734\"/\"auth-test-model-521734-kserve-self-signed-certs\"" Nov 01 22:19:36.784858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.784834 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-521734\"/\"default-dockercfg-cf727\"" Nov 01 22:19:36.867796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.865671 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:36.867796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.865761 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:36.869054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.869008 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.869346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.869312 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.870081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.870043 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.870247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.870227 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnql\" (UniqueName: \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.870424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.870396 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.870693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.870655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.951472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.933769 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:36.976138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.976063 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.995170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.995128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.995544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.995501 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.996273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.996250 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.996584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.996564 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.996772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.996735 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnql\" (UniqueName: \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:36.997192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:36.997171 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.018753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.018720 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.019559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.019536 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.022791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.022738 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.071633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.062976 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.076576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.074239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnql\" (UniqueName: \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\") pod \"auth-test-model-521734-kserve-78f6f58c8f-5hx7x\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.112915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.112861 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:37.210480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.204736 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:37.638488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.633947 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:19:37.726059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.726023 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:37.851406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.851025 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Nov 01 22:19:37.851406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:37.851079 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Nov 01 22:19:38.275652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.275605 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" event={"ID":"b7ef2efd-927a-43c1-83bf-805f880d0065","Type":"ContainerStarted","Data":"f11f913d5f015f617a71ce57604920dc38ef511ba0523a6b984d2edb7f14be83"} Nov 01 22:19:38.275831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.275671 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" event={"ID":"b7ef2efd-927a-43c1-83bf-805f880d0065","Type":"ContainerStarted","Data":"d12afbc5d96e24076f1a7bebb60c8b0d58266a3751fa0940935d10180cc5cef2"} Nov 01 22:19:38.303224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.303155 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" podStartSLOduration=2.303136265 podStartE2EDuration="2.303136265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:38.295359114 +0000 UTC m=+2131.988339601" watchObservedRunningTime="2026-11-01 22:19:38.303136265 +0000 UTC m=+2131.996116751" Nov 01 22:19:38.443354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.443309 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:38.853759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.853725 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:38.854225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.854181 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:38.854950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:38.854931 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:19:39.205270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:39.205164 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:39.294326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:39.294253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:39.944325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:39.944277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:40.650149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:40.650105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:40.866976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:40.866938 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:41.060758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.060719 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:41.141431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.141399 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:19:41.149966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.149939 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.163833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.163800 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:19:41.165031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.165005 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"openshift-service-ca.crt\"" Nov 01 22:19:41.173278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.165261 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"default-dockercfg-wcw8g\"" Nov 01 22:19:41.173505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.165331 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"sub-test-model-51e3b7-kserve-self-signed-certs\"" Nov 01 22:19:41.173646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.165399 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-51e3b7\"/\"kube-root-ca.crt\"" Nov 01 22:19:41.227463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.222760 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:19:41.227463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.223010 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" containerID="cri-o://6c50ff9f619edd413d0829c5633e6961d60cbcaf3cecc054611702c5f0509ef6" gracePeriod=30 Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.233768 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.233819 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.233886 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmws\" (UniqueName: \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.233987 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.234147 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.237466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.234206 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.250677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.248104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:41.336474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336361 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmws\" (UniqueName: \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.336681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336481 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.336681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336641 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.336812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336706 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.336903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336878 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.336976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.336922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.337150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.337121 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.337292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.337272 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.337944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.337920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.339985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.339830 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.339985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.339927 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.353115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.353087 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmws\" (UniqueName: \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\") pod \"sub-test-model-51e3b7-kserve-55c75999c5-2sghh\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.398219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.398021 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:41.497811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.497763 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:41.733466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.732109 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq"] Nov 01 22:19:41.795319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.792774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:41.842613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.840803 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:41.872146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.857930 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:19:41.872146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.863747 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:19:41.909460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.899661 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq"] Nov 01 22:19:41.964165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.964100 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrp9\" (UniqueName: \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:41.964318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.964299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:41.964435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:41.964419 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.067240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.067193 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrp9\" (UniqueName: \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.067240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.067236 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.067816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.067263 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.068536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.068511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.070578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.070554 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.082990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.080498 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrp9\" (UniqueName: \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\") pod \"maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.180014 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:19:42.179983 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62cd799_9ae1_4b43_8552_0902237db3a4.slice/crio-246a72653781f0a55c9e7108746e0f0108555d039e17d0aab6b6cf96779a8196 WatchSource:0}: Error finding container 246a72653781f0a55c9e7108746e0f0108555d039e17d0aab6b6cf96779a8196: Status 404 returned error can't find the container with id 246a72653781f0a55c9e7108746e0f0108555d039e17d0aab6b6cf96779a8196 Nov 01 22:19:42.182110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.182079 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:19:42.239406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.239380 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:42.374007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.373963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" event={"ID":"e62cd799-9ae1-4b43-8552-0902237db3a4","Type":"ContainerStarted","Data":"d9a5be8eb9b458fce8a79b60874a771aac5cacd2a53643d00fd0c543b65a0b2c"} Nov 01 22:19:42.374190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.374020 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" event={"ID":"e62cd799-9ae1-4b43-8552-0902237db3a4","Type":"ContainerStarted","Data":"246a72653781f0a55c9e7108746e0f0108555d039e17d0aab6b6cf96779a8196"} Nov 01 22:19:42.400520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.398593 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" podStartSLOduration=1.398574367 podStartE2EDuration="1.398574367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:42.394289979 +0000 UTC m=+2136.087270465" watchObservedRunningTime="2026-11-01 22:19:42.398574367 +0000 UTC m=+2136.091554855" Nov 01 22:19:42.498803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.498717 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:42.596090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.596039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq"] Nov 01 22:19:42.604147 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:19:42.604119 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec697bb1_dda1_4395_a5f5_ff10e9a4889c.slice/crio-e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709 WatchSource:0}: Error finding container e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709: Status 404 returned error can't find the container with id e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709 Nov 01 22:19:42.616105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:42.615987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:43.345845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.345806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:43.432129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.432089 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" event={"ID":"ec697bb1-dda1-4395-a5f5-ff10e9a4889c","Type":"ContainerStarted","Data":"f990e21182c555af2e35480a78f04ecd9f758b04478939fd312ed47de4bf0948"} Nov 01 22:19:43.432315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.432150 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" event={"ID":"ec697bb1-dda1-4395-a5f5-ff10e9a4889c","Type":"ContainerStarted","Data":"e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709"} Nov 01 22:19:43.457524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.457473 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" podStartSLOduration=2.457458954 podStartE2EDuration="2.457458954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:43.451771727 +0000 UTC m=+2137.144752213" watchObservedRunningTime="2026-11-01 22:19:43.457458954 +0000 UTC m=+2137.150439430" Nov 01 22:19:43.689152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.689068 5036 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:19:43.689152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:43.689127 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:19:44.059293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:44.059253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:44.442920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:44.442836 5036 generic.go:356] "Generic (PLEG): container finished" podID="ec697bb1-dda1-4395-a5f5-ff10e9a4889c" containerID="f990e21182c555af2e35480a78f04ecd9f758b04478939fd312ed47de4bf0948" exitCode=0 Nov 01 22:19:44.443299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:44.442919 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" event={"ID":"ec697bb1-dda1-4395-a5f5-ff10e9a4889c","Type":"ContainerDied","Data":"f990e21182c555af2e35480a78f04ecd9f758b04478939fd312ed47de4bf0948"} Nov 01 22:19:44.836714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:44.836678 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:45.632996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.632957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:45.639671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.639645 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:45.738637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.738549 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\") pod \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " Nov 01 22:19:45.738802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.738715 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\") pod \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " Nov 01 22:19:45.738860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.738848 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\" (UniqueName: \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\") pod \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\" (UID: \"ec697bb1-dda1-4395-a5f5-ff10e9a4889c\") " Nov 01 22:19:45.739569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.739537 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca" pod "ec697bb1-dda1-4395-a5f5-ff10e9a4889c" (UID: "ec697bb1-dda1-4395-a5f5-ff10e9a4889c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:45.739870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.739846 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:45.742629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.742605 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca" pod "ec697bb1-dda1-4395-a5f5-ff10e9a4889c" (UID: "ec697bb1-dda1-4395-a5f5-ff10e9a4889c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:45.742729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.742641 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9" pod "ec697bb1-dda1-4395-a5f5-ff10e9a4889c" (UID: "ec697bb1-dda1-4395-a5f5-ff10e9a4889c"). InnerVolumeSpecName "kube-api-access-gjrp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:45.841578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.841541 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:45.841578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:45.841580 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrp9\" (UniqueName: \"kubernetes.io/projected/ec697bb1-dda1-4395-a5f5-ff10e9a4889c-kube-api-access-gjrp9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.353949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.353900 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.477725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" event={"ID":"ec697bb1-dda1-4395-a5f5-ff10e9a4889c","Type":"ContainerDied","Data":"e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709"} Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.477879 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e847781c187d69d7d0681a6d42e60a4b0d4442474bcc4864485daef9bfaf6709" Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.477762 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq" Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.481813 5036 generic.go:356] "Generic (PLEG): container finished" podID="cec747a1-f425-4e2c-985b-292e1091fb29" containerID="6c50ff9f619edd413d0829c5633e6961d60cbcaf3cecc054611702c5f0509ef6" exitCode=0 Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.481843 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerDied","Data":"6c50ff9f619edd413d0829c5633e6961d60cbcaf3cecc054611702c5f0509ef6"} Nov 01 22:19:46.485504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.481930 5036 scope.go:122] "RemoveContainer" containerID="0d9a502f31be338a786c24f1ab65386182ab23afe4aa249971011f5c21e1dc9f" Nov 01 22:19:46.962537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:46.962509 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:19:47.064329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064287 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064395 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064527 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064624 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:47.064756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064707 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064776 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.064976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064920 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.065032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\") pod \"cec747a1-f425-4e2c-985b-292e1091fb29\" (UID: \"cec747a1-f425-4e2c-985b-292e1091fb29\") " Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.064997 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065194 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065322 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065615 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065900 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065928 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065945 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065959 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.066350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.065974 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec747a1-f425-4e2c-985b-292e1091fb29-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.068725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.068654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:47.068725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.068688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:19:47.068725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.068713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "kube-api-access-m9f8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:47.070329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.070307 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token" pod "cec747a1-f425-4e2c-985b-292e1091fb29" (UID: "cec747a1-f425-4e2c-985b-292e1091fb29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:47.083436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.083409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:47.167542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.167503 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec747a1-f425-4e2c-985b-292e1091fb29-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.167542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.167536 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9f8g\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-kube-api-access-m9f8g\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.167770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.167552 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec747a1-f425-4e2c-985b-292e1091fb29-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.167770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.167562 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec747a1-f425-4e2c-985b-292e1091fb29-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.206033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.205756 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:47.218877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.218850 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:47.495839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.495758 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" Nov 01 22:19:47.496015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.495845 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs" event={"ID":"cec747a1-f425-4e2c-985b-292e1091fb29","Type":"ContainerDied","Data":"a898b2fc6a1e52eb495f9ffb1c17a3748599301d855ebc1e44b09b1488d4e503"} Nov 01 22:19:47.496015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.495908 5036 scope.go:122] "RemoveContainer" containerID="6c50ff9f619edd413d0829c5633e6961d60cbcaf3cecc054611702c5f0509ef6" Nov 01 22:19:47.511567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.511542 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:19:47.554598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.554560 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:19:47.561670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.561639 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-dggzs"] Nov 01 22:19:47.835967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:47.835924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:48.440865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:48.440828 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" path="/var/lib/kubelet/pods/cec747a1-f425-4e2c-985b-292e1091fb29/volumes" Nov 01 22:19:48.547756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:48.547716 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:48.862485 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:19:48.862414 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:19:49.227479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:49.227385 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:49.949311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:49.949256 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:50.743632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:50.743588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:51.482205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:51.482160 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:51.501150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:51.501117 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:51.553140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:51.553107 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:51.571488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:51.571426 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:51.885153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:51.885117 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:19:52.106088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:52.106009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:52.259583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:52.259478 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:52.418495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:52.416091 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:53.252887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:53.252845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:54.032697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:54.032663 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:54.679768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:54.679722 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:55.101431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:55.101373 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:19:55.702434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:55.702401 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:56.391805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:56.391420 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:56.886569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:56.886509 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" podUID="a5092adf-ec66-4db1-b343-ef465a163e8c" containerName="payload-processing" containerID="cri-o://1d3fd27013759c6c48b3c4c97733c15b283c65b49bfa45cc1e1f28dbcd83449e" gracePeriod=25 Nov 01 22:19:57.565556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.565511 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:19:57.583112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.583076 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:19:57.583546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.583522 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:19:57.583784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.583770 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ec697bb1-dda1-4395-a5f5-ff10e9a4889c" containerName="revoke-keys" Nov 01 22:19:57.583899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.583887 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec697bb1-dda1-4395-a5f5-ff10e9a4889c" containerName="revoke-keys" Nov 01 22:19:57.584339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.584303 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.584639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.584219 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec697bb1-dda1-4395-a5f5-ff10e9a4889c" containerName="revoke-keys" Nov 01 22:19:57.584768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.584755 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:19:57.585373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.585357 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:19:57.586692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.585297 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.591490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.584860 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:19:57.606471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.593350 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" probeResult="failure" output="Get \"https://10.132.1.42:8000/ready\": context deadline exceeded" Nov 01 22:19:57.662467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.610283 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.662467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.613382 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:57.662467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.633263 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:57.717755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.668039 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"openshift-service-ca.crt\"" Nov 01 22:19:57.717755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.668486 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"default-dockercfg-zkjp8\"" Nov 01 22:19:57.717755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.668603 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"auth-test-model-d2832c-kserve-self-signed-certs\"" Nov 01 22:19:57.717755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.668705 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"kube-root-ca.crt\"" Nov 01 22:19:57.795486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.755519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.795486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.755741 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.795486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.783968 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": context deadline exceeded" Nov 01 22:19:57.828522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.800976 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.852796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.852757 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:19:57.878342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.856583 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:57.878342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.875632 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:19:57.885340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.882649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:19:57.923475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.907826 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:57.958773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.958742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:57.959213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.959185 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.161532 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66db\" (UniqueName: \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.161665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.161877 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:57.980866 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.126527 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.163599 5036 generic.go:356] "Generic (PLEG): container finished" podID="a5092adf-ec66-4db1-b343-ef465a163e8c" containerID="1d3fd27013759c6c48b3c4c97733c15b283c65b49bfa45cc1e1f28dbcd83449e" exitCode=0 Nov 01 22:19:58.182526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.163652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" event={"ID":"a5092adf-ec66-4db1-b343-ef465a163e8c","Type":"ContainerDied","Data":"1d3fd27013759c6c48b3c4c97733c15b283c65b49bfa45cc1e1f28dbcd83449e"} Nov 01 22:19:58.330464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.254423 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.365811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.276645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:58.366161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.318995 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:19:58.373519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.358700 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.386978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.359092 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.403737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.398210 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" probeResult="failure" output="Get \"https://10.132.1.20:8000/ready\": context deadline exceeded" Nov 01 22:19:58.453654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.453541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.453930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.453902 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66db\" (UniqueName: \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.565435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.531583 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.565435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.531821 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.565435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.531967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.565435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.532009 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.565435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.488534 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:19:58.566520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.566491 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.566691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.489003 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.576550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.567259 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.578322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.578278 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 01 22:19:58.578761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.578737 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded" Nov 01 22:19:58.626588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.607921 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.626588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.625556 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.627197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.627162 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": context deadline exceeded" Nov 01 22:19:58.647411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.647372 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.665326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.665266 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.665681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.665653 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": context deadline exceeded" Nov 01 22:19:58.666917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.666672 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" probeResult="failure" output="Get \"https://10.132.1.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721337 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721407 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721460 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721497 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721527 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:58.723681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.721556 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:19:58.793468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.785310 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.793468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.788746 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66db\" (UniqueName: \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.797466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.794088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.820163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.814714 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 01 22:19:58.820163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.816322 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:19:58.820163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.817473 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:19:58.820163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.817568 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:19:58.820163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.817649 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:19:58.876701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.876629 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:19:58.886628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.877385 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": context deadline exceeded" Nov 01 22:19:58.890133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.890103 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:19:58.894900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.894874 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:19:58.896992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.881624 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.917239 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.913976 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:19:58.945149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.942545 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\") pod \"auth-test-model-d2832c-kserve-586dcf5c49-qjw5m\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:58.945149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.944091 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.950477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:58.949259 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:19:59.204470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.179579 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:19:59.270841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.270808 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:19:59.324341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.324311 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:19:59.346462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.344166 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:59.380774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.376809 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:19:59.380774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.378870 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:19:59.403468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.394315 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:19:59.403468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.394387 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:19:59.403468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.394427 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:19:59.480575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.416938 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": context deadline exceeded" Nov 01 22:19:59.480575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.442491 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:19:59.480575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.442637 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:19:59.537479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.528942 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:19:59.537479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.529635 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:19:59.537479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.531998 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:19:59.537479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.532043 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:19:59.556483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.537934 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:19:59.556483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.541211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\") pod \"a5092adf-ec66-4db1-b343-ef465a163e8c\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " Nov 01 22:19:59.556483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.543115 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume" pod "a5092adf-ec66-4db1-b343-ef465a163e8c" (UID: "a5092adf-ec66-4db1-b343-ef465a163e8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:59.558148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.556873 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:19:59.558148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.557753 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:19:59.558148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.557801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:19:59.558475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.558201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:19:59.610005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.601038 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:19:59.610005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.603732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" event={"ID":"a5092adf-ec66-4db1-b343-ef465a163e8c","Type":"ContainerDied","Data":"9f4f96892822ddb9fa5bbdc59bb5a9fc90f6817a9eaf61fee18607cf2448888a"} Nov 01 22:19:59.610005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.607671 5036 scope.go:122] "RemoveContainer" containerID="1d3fd27013759c6c48b3c4c97733c15b283c65b49bfa45cc1e1f28dbcd83449e" Nov 01 22:19:59.614564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.614542 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf" Nov 01 22:19:59.634045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.634020 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:19:59.643051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.643020 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\" (UniqueName: \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\") pod \"a5092adf-ec66-4db1-b343-ef465a163e8c\" (UID: \"a5092adf-ec66-4db1-b343-ef465a163e8c\") " Nov 01 22:19:59.654573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.654529 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz" pod "a5092adf-ec66-4db1-b343-ef465a163e8c" (UID: "a5092adf-ec66-4db1-b343-ef465a163e8c"). InnerVolumeSpecName "kube-api-access-r4ftz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:59.669539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.657731 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5092adf-ec66-4db1-b343-ef465a163e8c-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:59.669539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.657762 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ftz\" (UniqueName: \"kubernetes.io/projected/a5092adf-ec66-4db1-b343-ef465a163e8c-kube-api-access-r4ftz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:59.669892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.669528 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:59.708949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.708620 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:19:59.976459 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:19:59.975987 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfefd7c6_8a4c_46a2_9be5_3e9a3d6349db.slice/crio-dff9b8c42c823f53ff80b8199ce8e570563dc4c6bc045569195d06868cab791d WatchSource:0}: Error finding container dff9b8c42c823f53ff80b8199ce8e570563dc4c6bc045569195d06868cab791d: Status 404 returned error can't find the container with id dff9b8c42c823f53ff80b8199ce8e570563dc4c6bc045569195d06868cab791d Nov 01 22:19:59.978243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:19:59.978074 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:20:00.067251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.067214 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:20:00.067917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.067491 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerName="maas-api" containerID="cri-o://0e93ff3ed15dd768ab39669029f33a2d0c1765023f2f82cb73f4959b88449cf2" gracePeriod=30 Nov 01 22:20:00.072608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.071394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:00.072608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.072308 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:20:00.090434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.089702 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-070e3d-6b97d99dd-8zshf"] Nov 01 22:20:00.212234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.212200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:00.359137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.350823 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.234:8443/health\": dial tcp 10.132.0.234:8443: connect: connection refused" Nov 01 22:20:00.375856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.375770 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:00.418785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.418716 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq"] Nov 01 22:20:00.581724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.581626 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:00.594335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.594287 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5092adf-ec66-4db1-b343-ef465a163e8c" path="/var/lib/kubelet/pods/a5092adf-ec66-4db1-b343-ef465a163e8c/volumes" Nov 01 22:20:00.602428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.602377 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-070e3d-52rcq"] Nov 01 22:20:00.798833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.798794 5036 generic.go:356] "Generic (PLEG): container finished" podID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerID="0e93ff3ed15dd768ab39669029f33a2d0c1765023f2f82cb73f4959b88449cf2" exitCode=0 Nov 01 22:20:00.799364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.798934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" event={"ID":"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7","Type":"ContainerDied","Data":"0e93ff3ed15dd768ab39669029f33a2d0c1765023f2f82cb73f4959b88449cf2"} Nov 01 22:20:00.802686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.802656 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:00.802793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.802719 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" event={"ID":"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db","Type":"ContainerStarted","Data":"21c84ec3f6b453215aa32cb8ee36432221d540e0fa2670a3f034107fab2e3794"} Nov 01 22:20:00.802793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.802748 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" event={"ID":"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db","Type":"ContainerStarted","Data":"dff9b8c42c823f53ff80b8199ce8e570563dc4c6bc045569195d06868cab791d"} Nov 01 22:20:00.844935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.844887 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" podStartSLOduration=3.84487335 podStartE2EDuration="3.84487335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:20:00.843049172 +0000 UTC m=+2154.536029660" watchObservedRunningTime="2026-11-01 22:20:00.84487335 +0000 UTC m=+2154.537853835" Nov 01 22:20:00.923349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:00.923319 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:20:01.032576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.032534 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\") pod \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " Nov 01 22:20:01.033413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.033383 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\") pod \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " Nov 01 22:20:01.033641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.033568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\" (UniqueName: \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\") pod \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " Nov 01 22:20:01.033928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.033911 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\") pod \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\" (UID: \"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7\") " Nov 01 22:20:01.037689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.037666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls" pod "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" (UID: "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:20:01.038218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.038196 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca" pod "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" (UID: "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:20:01.040994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.040967 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs" pod "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" (UID: "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:20:01.041501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.041473 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5" pod "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" (UID: "d93d83f2-2bcb-46ac-bb5e-6beee54eebb7"). InnerVolumeSpecName "kube-api-access-vx7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:01.141833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.141792 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:01.141833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.141833 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7h5\" (UniqueName: \"kubernetes.io/projected/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-kube-api-access-vx7h5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:01.142067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.141848 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:01.142067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.141861 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:01.423989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.423951 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:20:01.438987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.438948 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:01.815206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.815174 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" Nov 01 22:20:01.815414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.815211 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2" event={"ID":"d93d83f2-2bcb-46ac-bb5e-6beee54eebb7","Type":"ContainerDied","Data":"9df9eb4f3b347a660897ee64fc33a9e8dc1d25b210ad97cccb25ac3b3e0e2cac"} Nov 01 22:20:01.815414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.815280 5036 scope.go:122] "RemoveContainer" containerID="0e93ff3ed15dd768ab39669029f33a2d0c1765023f2f82cb73f4959b88449cf2" Nov 01 22:20:01.845155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.844790 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:20:01.850107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:01.850081 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-070e3d-64c7f44978-cltb2"] Nov 01 22:20:02.121218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:02.121102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:02.444551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:02.444459 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" path="/var/lib/kubelet/pods/d93d83f2-2bcb-46ac-bb5e-6beee54eebb7/volumes" Nov 01 22:20:02.445028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:02.445011 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec697bb1-dda1-4395-a5f5-ff10e9a4889c" path="/var/lib/kubelet/pods/ec697bb1-dda1-4395-a5f5-ff10e9a4889c/volumes" Nov 01 22:20:02.974767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:02.970759 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:03.673757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:03.673713 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:04.437974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:04.437932 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:05.176208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:05.176163 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:05.877078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:05.877045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:06.639393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:06.639359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:07.396282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:07.396226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:07.524082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:07.524044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:07.662416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:07.662329 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:20:07.663326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:07.663034 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" containerID="cri-o://ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310" gracePeriod=30 Nov 01 22:20:07.862468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:07.853646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:08.396777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:08.396734 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:09.135127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.135088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:09.424288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.424212 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:20:09.437555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.437524 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:20:09.767260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.767174 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 22:20:09.767433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.767411 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" containerID="cri-o://1396471fc8d520253932413bfeb99dac66090a27504d4588ef8171abd952b571" gracePeriod=30 Nov 01 22:20:09.844122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:09.844087 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:10.009785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.009754 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:20:10.025511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.025477 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64"] Nov 01 22:20:10.033923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.033885 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16fe198-b9s64"] Nov 01 22:20:10.333703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.333617 5036 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:20:10.333703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.333686 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:20:10.444640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.444601 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca9fcf4-76e0-491b-b852-b07cd569facb" path="/var/lib/kubelet/pods/0ca9fcf4-76e0-491b-b852-b07cd569facb/volumes" Nov 01 22:20:10.530365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:10.530326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:11.263682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:11.263630 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:11.949621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:11.949585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:12.713862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:12.713824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:12.997849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:12.997820 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:20:13.040300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040428 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040501 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040593 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040711 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040802 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040860 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.040907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040901 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.041029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.040966 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\") pod \"25e31560-3854-46f3-b8bb-aa04b5325eb9\" (UID: \"25e31560-3854-46f3-b8bb-aa04b5325eb9\") " Nov 01 22:20:13.041381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.041347 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.041629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.041603 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.041726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.041622 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.041801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.041727 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.042329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.042302 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.042429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.042335 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.042429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.042349 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.042429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.042361 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.042767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.042741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:20:13.046123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.046100 5036 generic.go:356] "Generic (PLEG): container finished" podID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerID="ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310" exitCode=0 Nov 01 22:20:13.046590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.046295 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerDied","Data":"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310"} Nov 01 22:20:13.046590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.046370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" event={"ID":"25e31560-3854-46f3-b8bb-aa04b5325eb9","Type":"ContainerDied","Data":"23e35d206c419d195690a7b56ce78531d9fce9707295d51b903f09c25bd30abe"} Nov 01 22:20:13.046590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.046403 5036 scope.go:122] "RemoveContainer" containerID="ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310" Nov 01 22:20:13.047361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.047342 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2" Nov 01 22:20:13.049338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.049300 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.049454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.049355 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "kube-api-access-rsp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:13.049454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.049386 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:13.050952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.050926 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo" pod "25e31560-3854-46f3-b8bb-aa04b5325eb9" (UID: "25e31560-3854-46f3-b8bb-aa04b5325eb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:20:13.083926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.083904 5036 scope.go:122] "RemoveContainer" containerID="c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488" Nov 01 22:20:13.097983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.097958 5036 scope.go:122] "RemoveContainer" containerID="ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310" Nov 01 22:20:13.098294 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:20:13.098269 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310\": container with ID starting with ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310 not found: ID does not exist" containerID="ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310" Nov 01 22:20:13.098391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.098308 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310"} err="failed to get container status \"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310\": rpc error: code = NotFound desc = could not find container \"ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310\": container with ID starting with ab80bd07c6c2b667a15c2760746ab8b14fd458d379be68da900c1e7491e5b310 not found: ID does not exist" Nov 01 22:20:13.098391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.098336 5036 scope.go:122] "RemoveContainer" containerID="c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488" Nov 01 22:20:13.098648 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:20:13.098630 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488\": container with ID starting with c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488 not found: ID does not exist" containerID="c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488" Nov 01 22:20:13.098710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.098656 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488"} err="failed to get container status \"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488\": rpc error: code = NotFound desc = could not find container \"c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488\": container with ID starting with c606b9493698bdbb4424970f13675165223ba02e5c4d31dd4b5e297441d3c488 not found: ID does not exist" Nov 01 22:20:13.144484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.144459 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.144484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.144483 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e31560-3854-46f3-b8bb-aa04b5325eb9-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.144666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.144494 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsp2l\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-kube-api-access-rsp2l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.144666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.144504 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.144666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.144514 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e31560-3854-46f3-b8bb-aa04b5325eb9-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.388693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.388648 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:20:13.402602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.402565 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-zsdg2"] Nov 01 22:20:13.621855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:13.621760 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:14.351557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:14.351508 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:14.441517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:14.441478 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" path="/var/lib/kubelet/pods/25e31560-3854-46f3-b8bb-aa04b5325eb9/volumes" Nov 01 22:20:14.513475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:14.509911 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:15.070949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.070884 5036 generic.go:356] "Generic (PLEG): container finished" podID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerID="1396471fc8d520253932413bfeb99dac66090a27504d4588ef8171abd952b571" exitCode=0 Nov 01 22:20:15.071089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.070974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" event={"ID":"5d9285de-c1a7-419e-bbb9-c083a10cf300","Type":"ContainerDied","Data":"1396471fc8d520253932413bfeb99dac66090a27504d4588ef8171abd952b571"} Nov 01 22:20:15.071089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.071079 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" event={"ID":"5d9285de-c1a7-419e-bbb9-c083a10cf300","Type":"ContainerDied","Data":"b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463"} Nov 01 22:20:15.071202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.071110 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b01992b49558c658a0723a420bb1d8ee6da339ddcb932e9119631fe422c71463" Nov 01 22:20:15.095090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.095060 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 22:20:15.134966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.134930 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:15.195678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.195641 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.195844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.195754 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.195844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.195804 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.195987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.195932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.195987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.195972 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:15.196113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196010 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.196172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196119 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.196172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196166 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196168 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196188 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196208 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196245 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\") pod \"5d9285de-c1a7-419e-bbb9-c083a10cf300\" (UID: \"5d9285de-c1a7-419e-bbb9-c083a10cf300\") " Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:20:15.197067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.196570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:15.197554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.197107 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.197554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.197133 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.197554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.197149 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.197554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.197165 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9285de-c1a7-419e-bbb9-c083a10cf300-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.197554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.197180 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.198647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.198552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:15.198806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.198733 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:15.198947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.198915 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "kube-api-access-h2v9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:15.199066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.198980 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo" pod "5d9285de-c1a7-419e-bbb9-c083a10cf300" (UID: "5d9285de-c1a7-419e-bbb9-c083a10cf300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:20:15.299165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.299070 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.299165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.299117 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.299165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.299134 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2v9j\" (UniqueName: \"kubernetes.io/projected/5d9285de-c1a7-419e-bbb9-c083a10cf300-kube-api-access-h2v9j\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.299165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.299150 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9285de-c1a7-419e-bbb9-c083a10cf300-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:15.971502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:15.971402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:16.081881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.081849 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9" Nov 01 22:20:16.113657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.113622 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 22:20:16.117844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.117819 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9sjh9"] Nov 01 22:20:16.440915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.440881 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" path="/var/lib/kubelet/pods/5d9285de-c1a7-419e-bbb9-c083a10cf300/volumes" Nov 01 22:20:16.728654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.728577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:16.769139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.769105 5036 scope.go:122] "RemoveContainer" containerID="66d9ebb5b16f8b974cc95d377d6a3e4ee4e7d7aa74cd370e4e47633246cb8c78" Nov 01 22:20:16.792868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.792838 5036 scope.go:122] "RemoveContainer" containerID="1396471fc8d520253932413bfeb99dac66090a27504d4588ef8171abd952b571" Nov 01 22:20:16.805432 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:16.805411 5036 scope.go:122] "RemoveContainer" containerID="4f94c45afa2afeb900cd54bfa85fe6f2137b307641dd90fd59d3a8ab5f4681d3" Nov 01 22:20:17.467540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:17.467507 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:18.244370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:18.244331 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:18.861657 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:20:18.861623 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:20:19.038614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:19.038574 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:19.762066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:19.762014 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:20.642422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:20.642388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:21.530056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:21.530012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:22.292395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:22.292348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:23.058561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:23.058529 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6"] Nov 01 22:20:23.065372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:23.065339 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad872dc3-mjxm6"] Nov 01 22:20:23.137702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:23.137666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:23.830309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:23.830253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:24.440043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:24.440005 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52dc55de-462e-44a5-a008-750cc7ff28f7" path="/var/lib/kubelet/pods/52dc55de-462e-44a5-a008-750cc7ff28f7/volumes" Nov 01 22:20:24.623324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:24.623283 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:25.441693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:25.441652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:26.149875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:26.149824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:26.826094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:26.826055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:27.039421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:27.039347 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z"] Nov 01 22:20:27.041718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:27.041684 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd730292-sn25z"] Nov 01 22:20:27.514822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:27.514785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:28.172766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:28.172729 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:28.440154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:28.440069 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749a065c-6d82-4094-a036-d47d93529fd2" path="/var/lib/kubelet/pods/749a065c-6d82-4094-a036-d47d93529fd2/volumes" Nov 01 22:20:28.841646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:28.841601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:29.528675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:29.528630 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:30.281097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.281058 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:30.510122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.510066 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.747536 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748740 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748763 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748794 5036 cpu_manager.go:414] "containerMap: removing container" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748801 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748814 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerName="maas-api" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748823 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerName="maas-api" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748856 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a5092adf-ec66-4db1-b343-ef465a163e8c" containerName="payload-processing" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748863 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5092adf-ec66-4db1-b343-ef465a163e8c" containerName="payload-processing" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748890 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.748897 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec747a1-f425-4e2c-985b-292e1091fb29" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.749089 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5092adf-ec66-4db1-b343-ef465a163e8c" containerName="payload-processing" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.749109 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.749126 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9285de-c1a7-419e-bbb9-c083a10cf300" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.749139 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:20:30.749601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.749147 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93d83f2-2bcb-46ac-bb5e-6beee54eebb7" containerName="maas-api" Nov 01 22:20:30.754281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.754230 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.760998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.760966 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-2c5135\"/\"openshift-service-ca.crt\"" Nov 01 22:20:30.763786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.763761 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-2c5135\"/\"kube-root-ca.crt\"" Nov 01 22:20:30.764662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.764637 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:20:30.768437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.768412 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-2c5135\"/\"rate-test-model-2c5135-kserve-self-signed-certs\"" Nov 01 22:20:30.768734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.768717 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-2c5135\"/\"default-dockercfg-bp8fv\"" Nov 01 22:20:30.813943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.813914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814306 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814483 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814646 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkrh\" (UniqueName: \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814703 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.814904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.814778 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:30.922577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.922534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.922788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.922611 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.922788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.922780 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkrh\" (UniqueName: \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.922910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.922834 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.922910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.922881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.923643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.923076 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.923801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.923770 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.924095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.924069 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.927840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.927783 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.931710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.931683 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.932637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.932610 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.941805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.941778 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkrh\" (UniqueName: \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\") pod \"rate-test-model-2c5135-kserve-5d5b597ddf-zw49t\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:30.949612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:30.949582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:31.087845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:31.087803 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:31.475708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:31.475585 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:20:31.480428 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:20:31.480390 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3be6be11_01b8_414b_9c36_1c1ab2a4a17b.slice/crio-427fc305596ac0c12e822bfc36bb99dccfcb6b41a24c982bab5c715d56cd558f WatchSource:0}: Error finding container 427fc305596ac0c12e822bfc36bb99dccfcb6b41a24c982bab5c715d56cd558f: Status 404 returned error can't find the container with id 427fc305596ac0c12e822bfc36bb99dccfcb6b41a24c982bab5c715d56cd558f Nov 01 22:20:31.756350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:31.756254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:32.277236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:32.277193 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" event={"ID":"3be6be11-01b8-414b-9c36-1c1ab2a4a17b","Type":"ContainerStarted","Data":"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8"} Nov 01 22:20:32.277487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:32.277247 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" event={"ID":"3be6be11-01b8-414b-9c36-1c1ab2a4a17b","Type":"ContainerStarted","Data":"427fc305596ac0c12e822bfc36bb99dccfcb6b41a24c982bab5c715d56cd558f"} Nov 01 22:20:32.297914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:32.297870 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" podStartSLOduration=2.297856955 podStartE2EDuration="2.297856955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:20:32.296495668 +0000 UTC m=+2185.989476155" watchObservedRunningTime="2026-11-01 22:20:32.297856955 +0000 UTC m=+2185.990837441" Nov 01 22:20:32.709514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:32.700055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:33.091420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:33.091383 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:33.765068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:33.765029 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:34.691890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:34.691852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:35.447887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:35.447847 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:36.339880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:36.339845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:37.030532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:37.030495 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh"] Nov 01 22:20:37.040613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:37.040578 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93c26cb-rlnmh"] Nov 01 22:20:37.196853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:37.196809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:38.040555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:38.040509 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:38.440617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:38.440524 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e9064d-fd73-4909-925c-70aa883720ac" path="/var/lib/kubelet/pods/60e9064d-fd73-4909-925c-70aa883720ac/volumes" Nov 01 22:20:38.936662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:38.936627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:39.821184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:39.821141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:40.645470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:40.645403 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:41.089525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:41.089439 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:41.102869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:41.102841 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:41.450230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:41.450125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:41.501465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:41.498766 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:20:42.150757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:42.150717 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:42.839016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:42.838979 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:43.552842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:43.552808 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:44.237174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:44.237132 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:44.916567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:44.916529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:45.756003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:45.755950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:46.442863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:46.442824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:47.130214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:47.130171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:47.869590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:47.869487 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:48.633327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:48.633287 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:48.862656 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:20:48.862621 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:20:49.290904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:49.290862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:50.057788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:50.057756 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:50.735243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:50.735193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:51.530793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:51.530753 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:52.166733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:52.166691 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:52.842857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:52.842814 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:53.535930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:53.535892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:54.220994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:54.220957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:55.021102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:55.021062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:55.745517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:55.745477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:56.436418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:56.436384 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:57.175987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:57.175945 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:57.851356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:57.851318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:58.542908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:58.542872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:59.295987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:59.295934 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:20:59.955003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:20:59.954966 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:00.152461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:00.152417 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:00.644728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:00.644683 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:01.185168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:01.185130 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:21:01.501969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:01.501878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:02.258710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:02.258675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:03.328778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:03.328743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:04.232353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.232315 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:04.314485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.311733 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:21:04.317425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.317389 5036 cpu_manager.go:414] "containerMap: removing container" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:21:04.317659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.317641 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e31560-3854-46f3-b8bb-aa04b5325eb9" containerName="istio-proxy" Nov 01 22:21:04.323712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.323679 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.331489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.328540 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:21:04.331985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.331956 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-521734-openshift-default-dockercfg-wmbjz\"" Nov 01 22:21:04.430005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.429970 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430108 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430140 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430321 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430561 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430540 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430664 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqz6b\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.430780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.430761 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.532662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.532613 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.532859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.532671 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.532859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.532778 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqz6b\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.532987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.532869 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533414 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533504 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533535 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533642 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.533847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.533831 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.534994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.534971 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.535316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.535288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.535423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.535400 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.535528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.535434 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.538812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.538767 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.539011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.538988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.546542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.546521 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqz6b\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.546944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.546919 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.656613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.656581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:04.955159 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:21:04.955123 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f9883_ac80_4467_a8a4_2d9009f84956.slice/crio-f4b4119305a27cb5cdb0c168038fb2bf7b02b9ff7e912b8ac54f6d6978283f6b WatchSource:0}: Error finding container f4b4119305a27cb5cdb0c168038fb2bf7b02b9ff7e912b8ac54f6d6978283f6b: Status 404 returned error can't find the container with id f4b4119305a27cb5cdb0c168038fb2bf7b02b9ff7e912b8ac54f6d6978283f6b Nov 01 22:21:04.955777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.955749 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:21:04.958785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.958745 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:04.958916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.958831 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:04.958916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:04.958876 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:05.119241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:05.110659 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:05.763050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:05.763012 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" event={"ID":"f40f9883-ac80-4467-a8a4-2d9009f84956","Type":"ContainerStarted","Data":"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8"} Nov 01 22:21:05.763508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:05.763068 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" event={"ID":"f40f9883-ac80-4467-a8a4-2d9009f84956","Type":"ContainerStarted","Data":"f4b4119305a27cb5cdb0c168038fb2bf7b02b9ff7e912b8ac54f6d6978283f6b"} Nov 01 22:21:05.813469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:05.812525 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podStartSLOduration=1.810846782 podStartE2EDuration="1.810846782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:05.794693751 +0000 UTC m=+2219.487674258" watchObservedRunningTime="2026-11-01 22:21:05.810846782 +0000 UTC m=+2219.503827269" Nov 01 22:21:05.860373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:05.860334 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:06.657025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:06.656995 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:06.658925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:06.658900 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body= Nov 01 22:21:06.659008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:06.658953 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" Nov 01 22:21:06.894468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:06.891957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:07.657354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:07.657314 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body= Nov 01 22:21:07.657674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:07.657379 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" Nov 01 22:21:07.895470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:07.894833 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:08.657101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:08.657062 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body= Nov 01 22:21:08.657410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:08.657126 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" Nov 01 22:21:08.801468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:08.800467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:09.643262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:09.643221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:09.666468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:09.662900 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body= Nov 01 22:21:09.666468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:09.662962 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" Nov 01 22:21:10.600496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:10.600457 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:10.658470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:10.657918 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body= Nov 01 22:21:10.658470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:10.657981 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" Nov 01 22:21:11.599197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:11.599156 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:11.671354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:11.671318 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:11.671830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:11.671714 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:11.673204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:11.673180 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:21:12.501795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:12.501756 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:13.084404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:13.084355 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr"] Nov 01 22:21:13.088560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:13.088529 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210b38cc-hdljr"] Nov 01 22:21:13.237900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:13.237860 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:14.080193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:14.080151 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:14.440732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:14.440654 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ce7343-48a7-40ec-923d-5ff711c836e7" path="/var/lib/kubelet/pods/11ce7343-48a7-40ec-923d-5ff711c836e7/volumes" Nov 01 22:21:14.725131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:14.725034 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:15.454340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:15.454303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:16.118105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:16.118062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:16.888659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:16.888589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:16.922777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:16.922695 5036 scope.go:122] "RemoveContainer" containerID="cededc50a3dc0954145707995ec3ec727685b31ce572f613846126ecc2029f41" Nov 01 22:21:17.642627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:17.642588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:18.331492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:18.331459 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:18.860112 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:21:18.860070 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:21:19.002650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:19.002606 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:19.729600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:19.729554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:20.441837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:20.441802 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:21.093906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:21.093866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:21.744225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:21.744189 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:22.433552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:22.433511 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:23.124098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:23.124056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:23.802789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:23.802736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:24.557685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:24.557647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:25.225507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:25.225425 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:26.037849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:26.037806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:26.771501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:26.771469 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:27.455895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:27.455851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:28.294508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:28.294469 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:28.883349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:28.883315 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:29.542541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:29.542505 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:30.318811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:30.318774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:30.957121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:30.957081 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:31.613612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:31.613566 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:32.351963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:32.351924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:33.040133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:33.040093 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:33.753708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:33.753660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:34.029035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:34.028996 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4"] Nov 01 22:21:34.033389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:34.033364 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea41e092-jkwr4"] Nov 01 22:21:34.441047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:34.440958 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b71d857-5d0e-45f3-ba93-18868b6b9371" path="/var/lib/kubelet/pods/0b71d857-5d0e-45f3-ba93-18868b6b9371/volumes" Nov 01 22:21:34.594965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:34.594928 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:35.235834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:35.235797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:35.987687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:35.987645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:36.715488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:36.715432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:37.445826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:37.445783 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:38.225324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:38.225274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:38.904984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:38.904935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:39.631967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:39.631923 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:40.339242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:40.339201 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:41.092538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:41.092494 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:41.750134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:41.750095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:42.543356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:42.543318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:43.181854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:43.181818 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:43.836925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:43.836889 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:44.648237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:44.648194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:45.438817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:45.438777 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:46.246688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:46.246654 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:47.047289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:47.047254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:47.738719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:47.738679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:48.430081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:48.430044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:48.860427 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:21:48.860388 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:21:49.091266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.091222 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:49.364342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.364304 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:21:49.371178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.371150 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.410119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.410080 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.432373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.432341 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.433062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.433035 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.433236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.433212 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.433406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.433321 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.436437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.436409 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.436607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.436552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.436607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.436588 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmdm\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.436800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.436653 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.447481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.447065 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:21:49.465654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.461119 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:21:49.499354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.486486 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:21:49.499354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.486581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550144 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550219 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550276 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550313 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550339 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550425 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550465 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550553 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mkj\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550586 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550601 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmdm\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550621 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550657 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.550694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550688 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.551133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550776 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.551133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550802 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.551133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550829 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.551133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.550893 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.551341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.551244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.552494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.551705 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.552494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.552074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.552763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.552742 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.553152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.553121 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.558468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.556979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.558468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.557159 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.566491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.565856 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.566491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.565975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmdm\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.652539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652427 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mkj\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652583 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652701 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652768 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.652830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652827 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.652921 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.653019 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.653227 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.653303 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.653330 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.653483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.653465 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.654887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.654865 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.655323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.655300 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.665152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.665131 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.665701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.665672 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mkj\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.785414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.782644 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:49.830710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.830675 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:49.939954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:49.939851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:50.105476 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:21:50.105418 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a733dee_efb9_4d83_be90_d6c17d8e1243.slice/crio-d786691b19b8864eafef7f1b1dbeeb304ab4dd272a7cb8e5d8e3bacec9f32247 WatchSource:0}: Error finding container d786691b19b8864eafef7f1b1dbeeb304ab4dd272a7cb8e5d8e3bacec9f32247: Status 404 returned error can't find the container with id d786691b19b8864eafef7f1b1dbeeb304ab4dd272a7cb8e5d8e3bacec9f32247 Nov 01 22:21:50.106263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.106235 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:21:50.110656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.110623 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.110775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.110704 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.110775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.110749 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.148163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.148139 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:21:50.150821 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:21:50.150797 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729ec215_cbdc_4eaf_b811_3dad6db7046c.slice/crio-809b42c3cd408e9d46a4771cfafe3c800cda1c947111a9990b7057ccafbea833 WatchSource:0}: Error finding container 809b42c3cd408e9d46a4771cfafe3c800cda1c947111a9990b7057ccafbea833: Status 404 returned error can't find the container with id 809b42c3cd408e9d46a4771cfafe3c800cda1c947111a9990b7057ccafbea833 Nov 01 22:21:50.153818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.153788 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.153899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.153852 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.153954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.153902 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:21:50.271479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.270189 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" event={"ID":"729ec215-cbdc-4eaf-b811-3dad6db7046c","Type":"ContainerStarted","Data":"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5"} Nov 01 22:21:50.271479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.270250 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" event={"ID":"729ec215-cbdc-4eaf-b811-3dad6db7046c","Type":"ContainerStarted","Data":"809b42c3cd408e9d46a4771cfafe3c800cda1c947111a9990b7057ccafbea833"} Nov 01 22:21:50.272685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.272651 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" event={"ID":"1a733dee-efb9-4d83-be90-d6c17d8e1243","Type":"ContainerStarted","Data":"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2"} Nov 01 22:21:50.272808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.272708 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" event={"ID":"1a733dee-efb9-4d83-be90-d6c17d8e1243","Type":"ContainerStarted","Data":"d786691b19b8864eafef7f1b1dbeeb304ab4dd272a7cb8e5d8e3bacec9f32247"} Nov 01 22:21:50.303883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.303813 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podStartSLOduration=1.303795471 podStartE2EDuration="1.303795471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:50.293306936 +0000 UTC m=+2263.986287423" watchObservedRunningTime="2026-11-01 22:21:50.303795471 +0000 UTC m=+2263.996775959" Nov 01 22:21:50.329166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.329095 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podStartSLOduration=1.329076365 podStartE2EDuration="1.329076365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:50.324166397 +0000 UTC m=+2264.017146885" watchObservedRunningTime="2026-11-01 22:21:50.329076365 +0000 UTC m=+2264.022056842" Nov 01 22:21:50.442253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.442121 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 22:21:50.442753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.442431 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" containerID="cri-o://0c5f0253ce543f34b12b69d6c38b776178a511c6edae07510a1b4f0a8a37d547" gracePeriod=30 Nov 01 22:21:50.785543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.785509 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:50.830979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.830944 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:50.847912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:50.847823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:51.705468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:51.700965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:51.794469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:51.792321 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:51.794469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:51.792382 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:51.854523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:51.854421 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:51.854523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:51.854516 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:52.554714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:52.548587 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:52.796425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:52.796391 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:52.796997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:52.796964 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:52.833531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:52.831551 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:52.833531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:52.831608 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:53.358218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.358166 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:53.565895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.565857 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:21:53.786478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.785750 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:53.786478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.785811 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:53.836468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.833874 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:53.836468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:53.833934 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:54.219540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:54.219432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:54.799468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:54.792940 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:54.799468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:54.793026 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:54.836495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:54.833992 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:54.836495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:54.834055 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:55.108628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.103621 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:55.789475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.786747 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:55.789475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.786808 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:55.836867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.836829 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:55.842520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.842475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:55.971560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:55.971373 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 22:21:56.121529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.120237 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 22:21:56.567470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.563425 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:21:56.683808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.680738 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:56.694367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.685143 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:56.947565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.881999 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:56.947565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.882062 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:56.988466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.987837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:57.001829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:56.990466 5036 generic.go:356] "Generic (PLEG): container finished" podID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerID="0c5f0253ce543f34b12b69d6c38b776178a511c6edae07510a1b4f0a8a37d547" exitCode=0 Nov 01 22:21:57.074703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.034974 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:57.074703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.035035 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:57.248474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.248347 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" event={"ID":"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7","Type":"ContainerDied","Data":"0c5f0253ce543f34b12b69d6c38b776178a511c6edae07510a1b4f0a8a37d547"} Nov 01 22:21:57.347819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.347757 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.349100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.349076 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 22:21:57.368218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.368180 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.453601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.453547 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.454666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.454635 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.472936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.472899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.489287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.489257 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.489631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.489596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.504743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.504667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.505541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.505516 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.506052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.506031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.506618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.506598 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.506819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.506801 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.516076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.505974 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:57.516303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.506521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:21:57.516416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.507737 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:57.516618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.516598 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\") pod \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\" (UID: \"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7\") " Nov 01 22:21:57.534472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.523302 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.575814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.575787 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.576044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.576029 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.576151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.574637 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.600469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.488146 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.45:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.534684 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.543559 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.603136 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.604537 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.604849 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.605084 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" probeResult="failure" output="Get \"https://10.132.1.41:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.605297 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" probeResult="failure" output="Get \"https://10.132.1.24:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.610505 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.611370 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.611636 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 22:21:57.615471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.611863 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" probeResult="failure" output="Get \"https://10.132.1.42:8000/ready\": context deadline exceeded" Nov 01 22:21:57.649247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.616551 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:21:57.649247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.633098 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.650921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.650867 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "kube-api-access-52f4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:57.651624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.651578 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:57.655462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.653652 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.684519 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": context deadline exceeded" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.687649 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": context deadline exceeded" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.688652 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.689517 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.689829 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.710523 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.756045 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.756082 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.756102 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52f4m\" (UniqueName: \"kubernetes.io/projected/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-kube-api-access-52f4m\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.756372 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.756394 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.763902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.759132 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:21:57.810508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.810471 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 01 22:21:57.812366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.812335 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 01 22:21:57.812574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.811887 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:21:57.846108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.831690 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 01 22:21:57.846108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.831740 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 01 22:21:57.846108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.843385 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo" pod "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" (UID: "7b4bcf45-5fd3-4874-a205-6a9e0c377aa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:21:57.861689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.861654 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:57.870609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.870568 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:57.955466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:57.923148 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:58.074899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.063908 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.38:8000/ready\": context deadline exceeded" Nov 01 22:21:58.075505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.073540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" probeResult="failure" output="Get \"https://10.132.1.44:8000/ready\": context deadline exceeded" Nov 01 22:21:58.097465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.074901 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 22:21:58.097840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.076801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:21:58.099628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.099599 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" event={"ID":"7b4bcf45-5fd3-4874-a205-6a9e0c377aa7","Type":"ContainerDied","Data":"fc367bca209319b14ba4519d999b8f686fa0b682b81db1138d75d316042efd6e"} Nov 01 22:21:58.099849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.099832 5036 scope.go:122] "RemoveContainer" containerID="0c5f0253ce543f34b12b69d6c38b776178a511c6edae07510a1b4f0a8a37d547" Nov 01 22:21:58.100089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.100071 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w" Nov 01 22:21:58.194545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.194501 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 22:21:58.200475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.199116 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-cjm6w"] Nov 01 22:21:58.442897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.442802 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" path="/var/lib/kubelet/pods/7b4bcf45-5fd3-4874-a205-6a9e0c377aa7/volumes" Nov 01 22:21:58.718208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.718108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:58.817468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.816647 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:58.817468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.817367 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:58.818669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.818645 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:21:58.837777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.837750 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:58.838069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.838047 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:58.838829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:58.838810 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:21:59.095260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.095209 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2"] Nov 01 22:21:59.095989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.095973 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" Nov 01 22:21:59.096043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.095990 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" Nov 01 22:21:59.096134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.096124 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4bcf45-5fd3-4874-a205-6a9e0c377aa7" containerName="istio-proxy" Nov 01 22:21:59.101469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.101420 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.106377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.106351 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:21:59.106996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.106976 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:21:59.123234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.119261 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2"] Nov 01 22:21:59.221702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.221656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.221702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.221701 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrs8p\" (UniqueName: \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.222311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.222279 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.323713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.323672 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.323713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.323713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrs8p\" (UniqueName: \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.324017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.323997 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.324322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.324294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.326487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.326461 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.336262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.336234 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrs8p\" (UniqueName: \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\") pod \"maas-api-revoke-sub-2de1f199-tcvk2\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.379197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.379120 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc"] Nov 01 22:21:59.384189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.384167 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.422272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.422232 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc"] Nov 01 22:21:59.425434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.425397 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.425646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.425534 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.425726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.425654 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77w5s\" (UniqueName: \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.435458 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.433311 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:21:59.539713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.539177 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.540035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.540012 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77w5s\" (UniqueName: \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.540497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.540430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.558191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.557904 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.565525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.565418 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.576530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.576224 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77w5s\" (UniqueName: \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\") pod \"maas-api-revoke-sub-c19aa3a2-xb5zc\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:21:59.607507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.607189 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:21:59.730487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.730411 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2"] Nov 01 22:21:59.731260 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:21:59.731229 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9586f0_1545_4678_ada0_550f745b70c0.slice/crio-8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c WatchSource:0}: Error finding container 8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c: Status 404 returned error can't find the container with id 8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c Nov 01 22:21:59.735324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:21:59.735208 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:22:00.042703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.042671 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc"] Nov 01 22:22:00.043388 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:22:00.043354 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b974f6_4023_4510_a5ee_72f3298c112e.slice/crio-9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8 WatchSource:0}: Error finding container 9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8: Status 404 returned error can't find the container with id 9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8 Nov 01 22:22:00.136801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.136763 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" event={"ID":"5d9586f0-1545-4678-ada0-550f745b70c0","Type":"ContainerStarted","Data":"8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c"} Nov 01 22:22:00.138529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.138500 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" event={"ID":"38b974f6-4023-4510-a5ee-72f3298c112e","Type":"ContainerStarted","Data":"9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8"} Nov 01 22:22:00.282788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.282743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:00.416045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.416006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:00.927591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:00.927493 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:01.150018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:01.149982 5036 generic.go:356] "Generic (PLEG): container finished" podID="38b974f6-4023-4510-a5ee-72f3298c112e" containerID="88a74be70b5e9e3c9233b9bf210974659126633e74d6a8e2c4cfde0d5a646079" exitCode=0 Nov 01 22:22:01.150510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:01.150078 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" event={"ID":"38b974f6-4023-4510-a5ee-72f3298c112e","Type":"ContainerDied","Data":"88a74be70b5e9e3c9233b9bf210974659126633e74d6a8e2c4cfde0d5a646079"} Nov 01 22:22:01.152082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:01.152061 5036 generic.go:356] "Generic (PLEG): container finished" podID="5d9586f0-1545-4678-ada0-550f745b70c0" containerID="ed5e47822ef660f4da6da6b4b72d5b0941383eac8245b3c5a8b3260091b80da4" exitCode=0 Nov 01 22:22:01.152199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:01.152094 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" event={"ID":"5d9586f0-1545-4678-ada0-550f745b70c0","Type":"ContainerDied","Data":"ed5e47822ef660f4da6da6b4b72d5b0941383eac8245b3c5a8b3260091b80da4"} Nov 01 22:22:01.649800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:01.649761 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:02.316405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.316369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:02.449243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.449163 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:22:02.512635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.510566 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:22:02.531085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.530004 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\" (UniqueName: \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\") pod \"38b974f6-4023-4510-a5ee-72f3298c112e\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " Nov 01 22:22:02.531085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.530057 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\") pod \"38b974f6-4023-4510-a5ee-72f3298c112e\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " Nov 01 22:22:02.531085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.530120 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\") pod \"38b974f6-4023-4510-a5ee-72f3298c112e\" (UID: \"38b974f6-4023-4510-a5ee-72f3298c112e\") " Nov 01 22:22:02.531754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.531161 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca" pod "38b974f6-4023-4510-a5ee-72f3298c112e" (UID: "38b974f6-4023-4510-a5ee-72f3298c112e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:02.541647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.541620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s" pod "38b974f6-4023-4510-a5ee-72f3298c112e" (UID: "38b974f6-4023-4510-a5ee-72f3298c112e"). InnerVolumeSpecName "kube-api-access-77w5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:02.551341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.551313 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca" pod "38b974f6-4023-4510-a5ee-72f3298c112e" (UID: "38b974f6-4023-4510-a5ee-72f3298c112e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:02.631916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.631881 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\") pod \"5d9586f0-1545-4678-ada0-550f745b70c0\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " Nov 01 22:22:02.632098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.631968 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\") pod \"5d9586f0-1545-4678-ada0-550f745b70c0\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " Nov 01 22:22:02.632098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632019 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\" (UniqueName: \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\") pod \"5d9586f0-1545-4678-ada0-550f745b70c0\" (UID: \"5d9586f0-1545-4678-ada0-550f745b70c0\") " Nov 01 22:22:02.632314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632291 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca" pod "5d9586f0-1545-4678-ada0-550f745b70c0" (UID: "5d9586f0-1545-4678-ada0-550f745b70c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:02.632841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632813 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.632965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632843 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77w5s\" (UniqueName: \"kubernetes.io/projected/38b974f6-4023-4510-a5ee-72f3298c112e-kube-api-access-77w5s\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.632965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632862 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.632965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.632879 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b974f6-4023-4510-a5ee-72f3298c112e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.634368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.634341 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca" pod "5d9586f0-1545-4678-ada0-550f745b70c0" (UID: "5d9586f0-1545-4678-ada0-550f745b70c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:02.634543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.634502 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p" pod "5d9586f0-1545-4678-ada0-550f745b70c0" (UID: "5d9586f0-1545-4678-ada0-550f745b70c0"). InnerVolumeSpecName "kube-api-access-jrs8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:02.734270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.734188 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9586f0-1545-4678-ada0-550f745b70c0-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.734270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.734221 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrs8p\" (UniqueName: \"kubernetes.io/projected/5d9586f0-1545-4678-ada0-550f745b70c0-kube-api-access-jrs8p\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:02.993770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:02.993660 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:03.173490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.173458 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" Nov 01 22:22:03.173490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.173473 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2" event={"ID":"5d9586f0-1545-4678-ada0-550f745b70c0","Type":"ContainerDied","Data":"8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c"} Nov 01 22:22:03.173762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.173537 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e8bb20e3987cace599a51846b51daf9da9010f9ca562eeeff6f63dd1c22e20c" Nov 01 22:22:03.175490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.175463 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" event={"ID":"38b974f6-4023-4510-a5ee-72f3298c112e","Type":"ContainerDied","Data":"9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8"} Nov 01 22:22:03.175613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.175506 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9789e4346338d2e3c35a9c77d06aea96d57177c4bb36c0a92c00093d3049f4d8" Nov 01 22:22:03.175613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.175515 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc" Nov 01 22:22:03.663102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:03.662996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:04.347617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:04.347578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:05.126133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:05.126096 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:05.824629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:05.824591 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:06.568468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:06.560634 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:07.244116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:07.244076 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:07.929934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:07.929895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:08.623341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:08.623301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:09.372429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.372395 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n"] Nov 01 22:22:09.373152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373135 5036 cpu_manager.go:414] "containerMap: removing container" podUID="38b974f6-4023-4510-a5ee-72f3298c112e" containerName="revoke-keys" Nov 01 22:22:09.373200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373154 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b974f6-4023-4510-a5ee-72f3298c112e" containerName="revoke-keys" Nov 01 22:22:09.373238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373210 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5d9586f0-1545-4678-ada0-550f745b70c0" containerName="revoke-keys" Nov 01 22:22:09.373238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373216 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9586f0-1545-4678-ada0-550f745b70c0" containerName="revoke-keys" Nov 01 22:22:09.373344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373334 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9586f0-1545-4678-ada0-550f745b70c0" containerName="revoke-keys" Nov 01 22:22:09.373378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.373349 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b974f6-4023-4510-a5ee-72f3298c112e" containerName="revoke-keys" Nov 01 22:22:09.378280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.378262 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.383135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.383110 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:22:09.394263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.393399 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n"] Nov 01 22:22:09.429025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.428989 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:09.469680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.468895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.469879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.469801 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrwh\" (UniqueName: \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.470176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.470156 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.485523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.483359 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:22:09.572291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.572260 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.572511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.572408 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.572589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.572523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrwh\" (UniqueName: \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.573071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.573041 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.574771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.574737 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.584854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.584828 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrwh\" (UniqueName: \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\") pod \"maas-api-revoke-sub-dba3aecf-bc74n\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.711782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.711697 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:09.788734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.788700 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8"] Nov 01 22:22:09.822487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.822431 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8"] Nov 01 22:22:09.822717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.822572 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.878092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.878055 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.878274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.878128 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.878346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.878266 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nfj\" (UniqueName: \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.980233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.980158 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.980398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.980245 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.980398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.980329 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nfj\" (UniqueName: \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.981108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.981076 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.982711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.982687 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:09.993223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:09.993149 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nfj\" (UniqueName: \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\") pod \"maas-api-revoke-sub-5d7f998f-766w8\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:10.002033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.002010 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n"] Nov 01 22:22:10.140384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.140356 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:10.260139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.260059 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" event={"ID":"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694","Type":"ContainerStarted","Data":"a7fc0b10d20dd9b461444fbe7f5a6684650d09ae88dab4250fb5089db85f21f8"} Nov 01 22:22:10.343935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.343894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:10.449919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.448927 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8"] Nov 01 22:22:10.454519 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:22:10.454488 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a8be8f_015e_4152_8885_81b244e35b78.slice/crio-94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173 WatchSource:0}: Error finding container 94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173: Status 404 returned error can't find the container with id 94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173 Nov 01 22:22:10.792693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.792668 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j"] Nov 01 22:22:10.797371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.797345 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:10.807269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.806375 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j"] Nov 01 22:22:10.910011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.909963 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmhc\" (UniqueName: \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:10.910188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.910148 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:10.910306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:10.910287 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.011956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.011868 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.012139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.011980 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.012339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.012224 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmhc\" (UniqueName: \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.012770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.012741 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.014706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.014674 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.024822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.024797 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmhc\" (UniqueName: \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\") pod \"maas-api-revoke-sub-beda212e-sqm9j\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.117511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.117480 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:11.141241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.141196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:11.273159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.273135 5036 generic.go:356] "Generic (PLEG): container finished" podID="ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" containerID="33a368d586fc6a731daa8387c7e2e986c79dbbfc1f1c355a8acb4f7557fecabf" exitCode=0 Nov 01 22:22:11.273369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.273353 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" event={"ID":"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694","Type":"ContainerDied","Data":"33a368d586fc6a731daa8387c7e2e986c79dbbfc1f1c355a8acb4f7557fecabf"} Nov 01 22:22:11.275884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.275865 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" event={"ID":"11a8be8f-015e-4152-8885-81b244e35b78","Type":"ContainerStarted","Data":"84d5f1555e6aaf049337bce3e14d7baf151104ceec98b8e154f215a3f066ad49"} Nov 01 22:22:11.276034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.276022 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" event={"ID":"11a8be8f-015e-4152-8885-81b244e35b78","Type":"ContainerStarted","Data":"94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173"} Nov 01 22:22:11.325536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.324515 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" podStartSLOduration=2.324497563 podStartE2EDuration="2.324497563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:11.320474161 +0000 UTC m=+2285.013454645" watchObservedRunningTime="2026-11-01 22:22:11.324497563 +0000 UTC m=+2285.017478050" Nov 01 22:22:11.485691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.485648 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j"] Nov 01 22:22:11.487029 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:22:11.486997 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5c986c_7e0c_426e_80e2_fcce17350802.slice/crio-6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff WatchSource:0}: Error finding container 6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff: Status 404 returned error can't find the container with id 6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff Nov 01 22:22:11.885052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:11.885009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:12.289332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.289290 5036 generic.go:356] "Generic (PLEG): container finished" podID="11a8be8f-015e-4152-8885-81b244e35b78" containerID="84d5f1555e6aaf049337bce3e14d7baf151104ceec98b8e154f215a3f066ad49" exitCode=0 Nov 01 22:22:12.289550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.289378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" event={"ID":"11a8be8f-015e-4152-8885-81b244e35b78","Type":"ContainerDied","Data":"84d5f1555e6aaf049337bce3e14d7baf151104ceec98b8e154f215a3f066ad49"} Nov 01 22:22:12.291520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.291486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" event={"ID":"8b5c986c-7e0c-426e-80e2-fcce17350802","Type":"ContainerStarted","Data":"91bc33acee3041ab600bdb6724b38454dfd90b39eaf83e63954c62480fd398e5"} Nov 01 22:22:12.291652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.291546 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" event={"ID":"8b5c986c-7e0c-426e-80e2-fcce17350802","Type":"ContainerStarted","Data":"6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff"} Nov 01 22:22:12.328872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.328810 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" podStartSLOduration=2.328794443 podStartE2EDuration="2.328794443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:12.325187259 +0000 UTC m=+2286.018167770" watchObservedRunningTime="2026-11-01 22:22:12.328794443 +0000 UTC m=+2286.021774933" Nov 01 22:22:12.488634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.488598 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:12.566264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.566170 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\") pod \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " Nov 01 22:22:12.566264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.566246 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\") pod \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " Nov 01 22:22:12.566568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.566277 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\" (UniqueName: \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\") pod \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\" (UID: \"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694\") " Nov 01 22:22:12.567089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.567057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca" pod "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" (UID: "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:12.567566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.567535 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:12.569421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.569396 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca" pod "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" (UID: "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:12.569604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.569584 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh" pod "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" (UID: "ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694"). InnerVolumeSpecName "kube-api-access-hrrwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:12.652379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.652347 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:12.682554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.669579 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:12.682554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:12.669618 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrwh\" (UniqueName: \"kubernetes.io/projected/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694-kube-api-access-hrrwh\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:13.305169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.305119 5036 generic.go:356] "Generic (PLEG): container finished" podID="8b5c986c-7e0c-426e-80e2-fcce17350802" containerID="91bc33acee3041ab600bdb6724b38454dfd90b39eaf83e63954c62480fd398e5" exitCode=0 Nov 01 22:22:13.305353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.305284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" event={"ID":"8b5c986c-7e0c-426e-80e2-fcce17350802","Type":"ContainerDied","Data":"91bc33acee3041ab600bdb6724b38454dfd90b39eaf83e63954c62480fd398e5"} Nov 01 22:22:13.307041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.307015 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" Nov 01 22:22:13.307189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.307074 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n" event={"ID":"ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694","Type":"ContainerDied","Data":"a7fc0b10d20dd9b461444fbe7f5a6684650d09ae88dab4250fb5089db85f21f8"} Nov 01 22:22:13.307189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.307134 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fc0b10d20dd9b461444fbe7f5a6684650d09ae88dab4250fb5089db85f21f8" Nov 01 22:22:13.336974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.336943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:13.579290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.579224 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:13.714755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.714722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\") pod \"11a8be8f-015e-4152-8885-81b244e35b78\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " Nov 01 22:22:13.714954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.714779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\" (UniqueName: \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\") pod \"11a8be8f-015e-4152-8885-81b244e35b78\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " Nov 01 22:22:13.715092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.715071 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\") pod \"11a8be8f-015e-4152-8885-81b244e35b78\" (UID: \"11a8be8f-015e-4152-8885-81b244e35b78\") " Nov 01 22:22:13.715506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.715431 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca" pod "11a8be8f-015e-4152-8885-81b244e35b78" (UID: "11a8be8f-015e-4152-8885-81b244e35b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:13.715778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.715758 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:13.717237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.717205 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj" pod "11a8be8f-015e-4152-8885-81b244e35b78" (UID: "11a8be8f-015e-4152-8885-81b244e35b78"). InnerVolumeSpecName "kube-api-access-g2nfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:13.717401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.717377 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca" pod "11a8be8f-015e-4152-8885-81b244e35b78" (UID: "11a8be8f-015e-4152-8885-81b244e35b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:13.817526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.817491 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8be8f-015e-4152-8885-81b244e35b78-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:13.817526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.817524 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nfj\" (UniqueName: \"kubernetes.io/projected/11a8be8f-015e-4152-8885-81b244e35b78-kube-api-access-g2nfj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:13.972201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:13.972120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:14.321328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.321300 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" Nov 01 22:22:14.321566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.321343 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8" event={"ID":"11a8be8f-015e-4152-8885-81b244e35b78","Type":"ContainerDied","Data":"94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173"} Nov 01 22:22:14.321566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.321405 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94904fbe0c3a2c18e73766cbea07cc30dbb95936cb7324da362c0dd676e1b173" Nov 01 22:22:14.502641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.502613 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:14.627153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.627064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:14.635502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.635471 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\") pod \"8b5c986c-7e0c-426e-80e2-fcce17350802\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " Nov 01 22:22:14.635693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.635578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\") pod \"8b5c986c-7e0c-426e-80e2-fcce17350802\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " Nov 01 22:22:14.635693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.635629 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\" (UniqueName: \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\") pod \"8b5c986c-7e0c-426e-80e2-fcce17350802\" (UID: \"8b5c986c-7e0c-426e-80e2-fcce17350802\") " Nov 01 22:22:14.640210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.638016 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca" pod "8b5c986c-7e0c-426e-80e2-fcce17350802" (UID: "8b5c986c-7e0c-426e-80e2-fcce17350802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:14.664466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.662823 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc" pod "8b5c986c-7e0c-426e-80e2-fcce17350802" (UID: "8b5c986c-7e0c-426e-80e2-fcce17350802"). InnerVolumeSpecName "kube-api-access-tdmhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:14.669864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.669827 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca" pod "8b5c986c-7e0c-426e-80e2-fcce17350802" (UID: "8b5c986c-7e0c-426e-80e2-fcce17350802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:14.738385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.738348 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:14.738385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.738384 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c986c-7e0c-426e-80e2-fcce17350802-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:14.738610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:14.738395 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmhc\" (UniqueName: \"kubernetes.io/projected/8b5c986c-7e0c-426e-80e2-fcce17350802-kube-api-access-tdmhc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:15.333495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:15.333464 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" Nov 01 22:22:15.333688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:15.333534 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j" event={"ID":"8b5c986c-7e0c-426e-80e2-fcce17350802","Type":"ContainerDied","Data":"6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff"} Nov 01 22:22:15.333688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:15.333642 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6729be1d3bf5a29dde187fe2b1a0c05731ab3b88633630b3bde0a526df7b27ff" Nov 01 22:22:15.356587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:15.356560 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:16.040388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:16.040343 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:16.689094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:16.689056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:16.995099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:16.995016 5036 scope.go:122] "RemoveContainer" containerID="63b9d214bbebd4152ac8749e2bb971d442ab0d060da48b33e3951b8e2d6fa1dc" Nov 01 22:22:17.008840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.008816 5036 scope.go:122] "RemoveContainer" containerID="4a0564de18b67f8e3772b2c2040b7b8d8acba7a88e5d037e287dbee3de6ab17d" Nov 01 22:22:17.021328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.021302 5036 scope.go:122] "RemoveContainer" containerID="cd4911b4d83b76ae68f01115c87a9eee708a60844e7c302f059d1ef606af090e" Nov 01 22:22:17.031788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.031760 5036 scope.go:122] "RemoveContainer" containerID="fe903b7f2c07cf1c935135ed0a8da50a4739d7742b2ffea20f0bc72cf3852908" Nov 01 22:22:17.042951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.042855 5036 scope.go:122] "RemoveContainer" containerID="e6101e5b9fd4530c56441a10b4d202c99bea9687b07c41230a970c4d2030e575" Nov 01 22:22:17.052915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.052890 5036 scope.go:122] "RemoveContainer" containerID="920a517732a13ae43ebc30680f3e48ab2ec13c82224ec0427e748e947d26e4cc" Nov 01 22:22:17.062352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.062329 5036 scope.go:122] "RemoveContainer" containerID="d1c2e81750090c49c2b4140faca4b3b2e111f2f0109e4dfb8cb48440a25eddd5" Nov 01 22:22:17.071849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.071833 5036 scope.go:122] "RemoveContainer" containerID="5da209411034c8219879055d1d20d48c3096666f117e3f77f2fcdece009bd04b" Nov 01 22:22:17.097605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.088594 5036 scope.go:122] "RemoveContainer" containerID="ed2a6862eac116026553b3efe6c6a711bacd4f0f3148f26053278a1a0075386e" Nov 01 22:22:17.122235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.122215 5036 scope.go:122] "RemoveContainer" containerID="c1cb896eb4f3857e298b2f768cc8ee413b0c658f2422da834fa3bed7731f1639" Nov 01 22:22:17.425955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:17.425882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:18.224790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:18.224751 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:18.860199 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:18.860168 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:22:18.878869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:18.878826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:19.632329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:19.632289 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:20.484337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.484297 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:20.540189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.539140 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k"] Nov 01 22:22:20.540409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.540338 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" containerName="revoke-keys" Nov 01 22:22:20.540409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.540360 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" containerName="revoke-keys" Nov 01 22:22:20.540409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.540393 5036 cpu_manager.go:414] "containerMap: removing container" podUID="11a8be8f-015e-4152-8885-81b244e35b78" containerName="revoke-keys" Nov 01 22:22:20.540409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.540402 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a8be8f-015e-4152-8885-81b244e35b78" containerName="revoke-keys" Nov 01 22:22:20.563534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.563498 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8b5c986c-7e0c-426e-80e2-fcce17350802" containerName="revoke-keys" Nov 01 22:22:20.563770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.563748 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5c986c-7e0c-426e-80e2-fcce17350802" containerName="revoke-keys" Nov 01 22:22:20.564307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.564281 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" containerName="revoke-keys" Nov 01 22:22:20.564484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.564468 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a8be8f-015e-4152-8885-81b244e35b78" containerName="revoke-keys" Nov 01 22:22:20.564590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.564580 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5c986c-7e0c-426e-80e2-fcce17350802" containerName="revoke-keys" Nov 01 22:22:20.605575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.605535 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k"] Nov 01 22:22:20.605868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.605854 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.611090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.611065 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:22:20.616304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.616282 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:22:20.674692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.674660 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkm5\" (UniqueName: \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.675103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.674925 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.675103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.675064 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.776339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.776298 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkm5\" (UniqueName: \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.777550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.776557 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.777550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.776678 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.777550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.777427 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.779533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.779511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.788535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.788510 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkm5\" (UniqueName: \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\") pod \"maas-api-revoke-sub-2ef8c9cd-mmp6k\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.936004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.935975 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:20.987510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.987406 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz"] Nov 01 22:22:20.993522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:20.993495 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.004471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.002530 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz"] Nov 01 22:22:21.084696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.084581 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854b5\" (UniqueName: \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.084696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.084650 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.084944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.084923 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.186461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.186400 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854b5\" (UniqueName: \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.186681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.186509 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.186808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.186788 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.187472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.187430 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.189492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.189428 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.203920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.203888 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854b5\" (UniqueName: \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\") pod \"maas-api-revoke-sub-0c126bc0-h4szz\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.248193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.248158 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:21.256550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.256475 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k"] Nov 01 22:22:21.282014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.281982 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:22:21.324199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.322842 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:21.387732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.387592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:21.563727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.563684 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" event={"ID":"c40daf00-247f-48e8-82b5-8f73e6c4a8d3","Type":"ContainerStarted","Data":"cc4ccb8b2895a4876a91aa2f4531fcdb785b3d4c33605f072aeaf3d8b61abbb2"} Nov 01 22:22:21.882402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:21.881858 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz"] Nov 01 22:22:22.040616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.040579 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:22.531771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.531713 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:22.574329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.574252 5036 generic.go:356] "Generic (PLEG): container finished" podID="c40daf00-247f-48e8-82b5-8f73e6c4a8d3" containerID="46ab25c3f09d9a4cb6b8286d0b8ec8408dd9585e7338551e0015438afe60dc97" exitCode=0 Nov 01 22:22:22.574329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.574306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" event={"ID":"c40daf00-247f-48e8-82b5-8f73e6c4a8d3","Type":"ContainerDied","Data":"46ab25c3f09d9a4cb6b8286d0b8ec8408dd9585e7338551e0015438afe60dc97"} Nov 01 22:22:22.576378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.576356 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" event={"ID":"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19","Type":"ContainerStarted","Data":"a217ffffa71842b02c1d02dc84964bcb0edbb72e7cc2df75295a0bff6d7c0d69"} Nov 01 22:22:22.576509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.576396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" event={"ID":"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19","Type":"ContainerStarted","Data":"44d04eec79a115275877140e55f87b1ff419e7ae17c40344c23b22f37a0a0200"} Nov 01 22:22:22.625552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.625496 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" podStartSLOduration=2.625480007 podStartE2EDuration="2.625480007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:22.618688583 +0000 UTC m=+2296.311669078" watchObservedRunningTime="2026-11-01 22:22:22.625480007 +0000 UTC m=+2296.318460494" Nov 01 22:22:22.751182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:22.751145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.561082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:23.561042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.650514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:23.649881 5036 generic.go:356] "Generic (PLEG): container finished" podID="19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" containerID="a217ffffa71842b02c1d02dc84964bcb0edbb72e7cc2df75295a0bff6d7c0d69" exitCode=0 Nov 01 22:22:23.650741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:23.650544 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" event={"ID":"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19","Type":"ContainerDied","Data":"a217ffffa71842b02c1d02dc84964bcb0edbb72e7cc2df75295a0bff6d7c0d69"} Nov 01 22:22:23.871181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:23.871158 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:24.043204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.043160 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\") pod \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " Nov 01 22:22:24.043370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.043332 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\" (UniqueName: \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\") pod \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " Nov 01 22:22:24.043423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.043368 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\") pod \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\" (UID: \"c40daf00-247f-48e8-82b5-8f73e6c4a8d3\") " Nov 01 22:22:24.043804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.043769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca" pod "c40daf00-247f-48e8-82b5-8f73e6c4a8d3" (UID: "c40daf00-247f-48e8-82b5-8f73e6c4a8d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:24.044175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.044154 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:24.045455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.045412 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca" pod "c40daf00-247f-48e8-82b5-8f73e6c4a8d3" (UID: "c40daf00-247f-48e8-82b5-8f73e6c4a8d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:24.045455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.045426 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5" pod "c40daf00-247f-48e8-82b5-8f73e6c4a8d3" (UID: "c40daf00-247f-48e8-82b5-8f73e6c4a8d3"). InnerVolumeSpecName "kube-api-access-whkm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:24.146437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.146334 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:24.146437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.146371 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkm5\" (UniqueName: \"kubernetes.io/projected/c40daf00-247f-48e8-82b5-8f73e6c4a8d3-kube-api-access-whkm5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:24.406152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.406070 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:24.659930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.659845 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" Nov 01 22:22:24.659930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.659892 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k" event={"ID":"c40daf00-247f-48e8-82b5-8f73e6c4a8d3","Type":"ContainerDied","Data":"cc4ccb8b2895a4876a91aa2f4531fcdb785b3d4c33605f072aeaf3d8b61abbb2"} Nov 01 22:22:24.660418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.659942 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc4ccb8b2895a4876a91aa2f4531fcdb785b3d4c33605f072aeaf3d8b61abbb2" Nov 01 22:22:24.864139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.864112 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:24.968144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.968058 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\") pod \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " Nov 01 22:22:24.968327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.968264 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\") pod \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " Nov 01 22:22:24.968327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.968293 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\" (UniqueName: \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\") pod \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\" (UID: \"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19\") " Nov 01 22:22:24.968552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.968527 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca" pod "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" (UID: "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:24.968995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.968960 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:24.970518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.970493 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5" pod "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" (UID: "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19"). InnerVolumeSpecName "kube-api-access-854b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:24.970603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:24.970516 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca" pod "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" (UID: "19cd34dc-7c46-4a3b-b953-2cfe5f1bba19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:25.070738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.070698 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:25.070738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.070727 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854b5\" (UniqueName: \"kubernetes.io/projected/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19-kube-api-access-854b5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:25.416120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.416081 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:25.670822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.670739 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" Nov 01 22:22:25.670822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.670780 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz" event={"ID":"19cd34dc-7c46-4a3b-b953-2cfe5f1bba19","Type":"ContainerDied","Data":"44d04eec79a115275877140e55f87b1ff419e7ae17c40344c23b22f37a0a0200"} Nov 01 22:22:25.671360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:25.670836 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d04eec79a115275877140e55f87b1ff419e7ae17c40344c23b22f37a0a0200" Nov 01 22:22:26.275021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:26.274987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:27.437006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:27.436968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:28.202296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:28.202258 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:22:28.227145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:28.227082 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:28.669996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:28.669958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:29.440550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:29.440516 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:30.253233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:30.253195 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:22:30.423480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:30.423429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:30.553849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:30.553810 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:30.897829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:30.897734 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:31.260259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:31.259472 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:31.478691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:31.478651 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:31.961073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:31.961030 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:32.142019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:32.141975 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:32.931121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:32.931083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:33.798964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:33.798929 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:34.561867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:34.561829 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:34.703494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:34.703395 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:35.547017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:35.546967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:35.823496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:35.823400 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:36.024538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.024483 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:36.314276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.314219 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" containerID="cri-o://6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192" gracePeriod=45 Nov 01 22:22:36.394832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.394794 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.17:8000/ready\": dial tcp 10.132.1.17:8000: connect: connection refused" Nov 01 22:22:36.648412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.648379 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:22:36.669291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.669261 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.669499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.669321 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\" (UniqueName: \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.669499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.669402 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.669499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.669434 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.669499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.669476 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.670147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.670117 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:36.670147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.670136 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:36.670365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.670345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\") pod \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\" (UID: \"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d\") " Nov 01 22:22:36.671143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.670934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:36.671514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.671495 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.671683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.671518 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.671683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.671531 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.673558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.673522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:36.673734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.673708 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:36.673882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.673856 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd" pod "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" (UID: "76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d"). InnerVolumeSpecName "kube-api-access-8tccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:36.773986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.773950 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.773986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.773988 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tccd\" (UniqueName: \"kubernetes.io/projected/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-kube-api-access-8tccd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.774203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.774002 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:36.849229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.849189 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:36.876532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.860740 5036 generic.go:356] "Generic (PLEG): container finished" podID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerID="6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192" exitCode=0 Nov 01 22:22:36.876532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.860969 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" event={"ID":"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d","Type":"ContainerDied","Data":"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192"} Nov 01 22:22:36.876532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.861019 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" event={"ID":"76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d","Type":"ContainerDied","Data":"eb385f514a0b1ef025aad61aeb81404d18d3aeaa8f2b4012467d6bedc5679f45"} Nov 01 22:22:36.876532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.861042 5036 scope.go:122] "RemoveContainer" containerID="6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192" Nov 01 22:22:36.876532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.861218 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv" Nov 01 22:22:36.952540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.952518 5036 scope.go:122] "RemoveContainer" containerID="6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192" Nov 01 22:22:36.952988 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:36.952948 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192\": container with ID starting with 6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192 not found: ID does not exist" containerID="6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192" Nov 01 22:22:36.953131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.952996 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192"} err="failed to get container status \"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192\": rpc error: code = NotFound desc = could not find container \"6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192\": container with ID starting with 6877f11adaa4b1de1df9a7ceec3a727df991303395a7a850d08b77e96521e192 not found: ID does not exist" Nov 01 22:22:36.958776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.958747 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:22:36.966903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:36.966879 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d6e25f/auth-test-model-d6e25f-kserve-576c87d997-blvbv"] Nov 01 22:22:37.712271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:37.712232 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:38.422229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:38.422163 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:38.459842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:38.459782 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" path="/var/lib/kubelet/pods/76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d/volumes" Nov 01 22:22:38.460333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:38.460305 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:22:38.569990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:38.569905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:38.865917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:38.865880 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:22:39.010254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:39.010206 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:39.307467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:39.307058 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:40.249005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:40.248963 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:41.039467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.039390 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:22:41.049948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.049907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:41.199470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.199122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:41.516110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.516024 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7"] Nov 01 22:22:41.517030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517003 5036 cpu_manager.go:414] "containerMap: removing container" podUID="19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" containerName="revoke-keys" Nov 01 22:22:41.517030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517030 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" containerName="revoke-keys" Nov 01 22:22:41.517223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517050 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c40daf00-247f-48e8-82b5-8f73e6c4a8d3" containerName="revoke-keys" Nov 01 22:22:41.517223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517059 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40daf00-247f-48e8-82b5-8f73e6c4a8d3" containerName="revoke-keys" Nov 01 22:22:41.517223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517071 5036 cpu_manager.go:414] "containerMap: removing container" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" Nov 01 22:22:41.517223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517081 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" Nov 01 22:22:41.517464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517272 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" containerName="revoke-keys" Nov 01 22:22:41.517464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517286 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40daf00-247f-48e8-82b5-8f73e6c4a8d3" containerName="revoke-keys" Nov 01 22:22:41.517464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.517297 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="76aa2e8d-97d7-41b1-8eec-056f6ccd5c6d" containerName="main" Nov 01 22:22:41.523481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.522683 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.529806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.529778 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:22:41.529983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.529966 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:22:41.531495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.531473 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7"] Nov 01 22:22:41.669268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.669229 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.669268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.669270 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwfx\" (UniqueName: \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.669522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.669390 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.771119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.771052 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.771346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.771200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwfx\" (UniqueName: \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.771346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.771297 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.771993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.771971 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.773567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.773537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.784203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.784182 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwfx\" (UniqueName: \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\") pod \"maas-api-revoke-sub-533654c4-n4pv7\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.859702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.859668 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:41.942494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:41.939687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:42.158768 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:22:42.158737 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24eccbe5_cd28_4979_b9aa_2e2112acc2a4.slice/crio-d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596 WatchSource:0}: Error finding container d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596: Status 404 returned error can't find the container with id d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596 Nov 01 22:22:42.158913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:42.158823 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7"] Nov 01 22:22:42.650434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:42.650393 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:43.075126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.075084 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" event={"ID":"24eccbe5-cd28-4979-b9aa-2e2112acc2a4","Type":"ContainerStarted","Data":"774cf26014c657e6a89a22a41123467c7ae090722e0983af848420b04b52b3df"} Nov 01 22:22:43.075306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.075153 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" event={"ID":"24eccbe5-cd28-4979-b9aa-2e2112acc2a4","Type":"ContainerStarted","Data":"d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596"} Nov 01 22:22:43.099365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.094144 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" podStartSLOduration=2.094125706 podStartE2EDuration="2.094125706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:43.093052991 +0000 UTC m=+2316.786033479" watchObservedRunningTime="2026-11-01 22:22:43.094125706 +0000 UTC m=+2316.787106193" Nov 01 22:22:43.233197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.233160 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" containerID="cri-o://005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127" gracePeriod=45 Nov 01 22:22:43.387174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.387137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:43.625331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.625300 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:22:43.653157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653126 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653168 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\" (UniqueName: \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653353 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653389 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:43.653836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\") pod \"9532214b-a852-4158-8d4b-6543bd7cac1d\" (UID: \"9532214b-a852-4158-8d4b-6543bd7cac1d\") " Nov 01 22:22:43.653836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653726 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:43.654006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.653928 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:43.654359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.654332 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:43.654468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.654366 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:43.654468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.654382 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:43.655875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.655845 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:43.655993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.655949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "kube-api-access-r2grp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:43.655993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.655970 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs" pod "9532214b-a852-4158-8d4b-6543bd7cac1d" (UID: "9532214b-a852-4158-8d4b-6543bd7cac1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:43.756371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.756334 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9532214b-a852-4158-8d4b-6543bd7cac1d-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:43.756371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.756372 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9532214b-a852-4158-8d4b-6543bd7cac1d-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:43.756580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:43.756388 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2grp\" (UniqueName: \"kubernetes.io/projected/9532214b-a852-4158-8d4b-6543bd7cac1d-kube-api-access-r2grp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:44.088740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.088688 5036 generic.go:356] "Generic (PLEG): container finished" podID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerID="005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127" exitCode=0 Nov 01 22:22:44.088740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.088738 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" Nov 01 22:22:44.088995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.088974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" event={"ID":"9532214b-a852-4158-8d4b-6543bd7cac1d","Type":"ContainerDied","Data":"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127"} Nov 01 22:22:44.089061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.089013 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md" event={"ID":"9532214b-a852-4158-8d4b-6543bd7cac1d","Type":"ContainerDied","Data":"7c131a147186ca98e2dc51b61fd9d745a2b07ec8ed7ddb005c5aa436c24d9cde"} Nov 01 22:22:44.089061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.089039 5036 scope.go:122] "RemoveContainer" containerID="005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127" Nov 01 22:22:44.094688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.094658 5036 generic.go:356] "Generic (PLEG): container finished" podID="24eccbe5-cd28-4979-b9aa-2e2112acc2a4" containerID="774cf26014c657e6a89a22a41123467c7ae090722e0983af848420b04b52b3df" exitCode=0 Nov 01 22:22:44.094826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.094706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" event={"ID":"24eccbe5-cd28-4979-b9aa-2e2112acc2a4","Type":"ContainerDied","Data":"774cf26014c657e6a89a22a41123467c7ae090722e0983af848420b04b52b3df"} Nov 01 22:22:44.106377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.105438 5036 scope.go:122] "RemoveContainer" containerID="005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127" Nov 01 22:22:44.106774 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:44.106749 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127\": container with ID starting with 005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127 not found: ID does not exist" containerID="005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127" Nov 01 22:22:44.106907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.106785 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127"} err="failed to get container status \"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127\": rpc error: code = NotFound desc = could not find container \"005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127\": container with ID starting with 005758bac9fd279bcb625e63c7110802faf5559897a46d232ddfb1e2bfc80127 not found: ID does not exist" Nov 01 22:22:44.129424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.129395 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:44.136039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.136016 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:22:44.163132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.144209 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/auth-test-model-1783e3-kserve-f7cb94fbf-st4md"] Nov 01 22:22:44.441418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.441342 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" path="/var/lib/kubelet/pods/9532214b-a852-4158-8d4b-6543bd7cac1d/volumes" Nov 01 22:22:44.859880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:44.859843 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:45.299075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.299032 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:45.299215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.299032 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" containerID="cri-o://57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d" gracePeriod=45 Nov 01 22:22:45.463540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.463464 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\") pod \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " Nov 01 22:22:45.463763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.463742 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\") pod \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " Nov 01 22:22:45.463856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.463789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\" (UniqueName: \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\") pod \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\" (UID: \"24eccbe5-cd28-4979-b9aa-2e2112acc2a4\") " Nov 01 22:22:45.463856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.463795 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca" pod "24eccbe5-cd28-4979-b9aa-2e2112acc2a4" (UID: "24eccbe5-cd28-4979-b9aa-2e2112acc2a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:22:45.464717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.464691 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.466029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.466000 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca" pod "24eccbe5-cd28-4979-b9aa-2e2112acc2a4" (UID: "24eccbe5-cd28-4979-b9aa-2e2112acc2a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:45.466357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.466330 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx" pod "24eccbe5-cd28-4979-b9aa-2e2112acc2a4" (UID: "24eccbe5-cd28-4979-b9aa-2e2112acc2a4"). InnerVolumeSpecName "kube-api-access-8dwfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:45.566289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.566254 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.566289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.566288 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwfx\" (UniqueName: \"kubernetes.io/projected/24eccbe5-cd28-4979-b9aa-2e2112acc2a4-kube-api-access-8dwfx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.590089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.590057 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:22:45.645633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.645594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:45.710318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710284 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.710567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710515 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.710567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710560 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\" (UniqueName: \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.710764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710622 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.710764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710672 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.710764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.710749 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\") pod \"2fae7803-f768-4c60-8328-abe5665107e0\" (UID: \"2fae7803-f768-4c60-8328-abe5665107e0\") " Nov 01 22:22:45.711311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.711282 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:45.711543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.711523 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.711707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.711685 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:45.711784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.711707 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:45.719731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.719664 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:45.733989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.733950 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:45.735944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.735894 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt" pod "2fae7803-f768-4c60-8328-abe5665107e0" (UID: "2fae7803-f768-4c60-8328-abe5665107e0"). InnerVolumeSpecName "kube-api-access-9s7zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:45.813208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.813176 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fae7803-f768-4c60-8328-abe5665107e0-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.813208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.813204 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7zt\" (UniqueName: \"kubernetes.io/projected/2fae7803-f768-4c60-8328-abe5665107e0-kube-api-access-9s7zt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.813208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.813217 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.813579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.813227 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:45.813579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:45.813235 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fae7803-f768-4c60-8328-abe5665107e0-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:46.121041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.121009 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" Nov 01 22:22:46.121041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.121031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7" event={"ID":"24eccbe5-cd28-4979-b9aa-2e2112acc2a4","Type":"ContainerDied","Data":"d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596"} Nov 01 22:22:46.121691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.121089 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ddc5648c7bd5a01898b1a03d40928b1fc761fc6bb44b242953ce28cb309596" Nov 01 22:22:46.122976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.122946 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fae7803-f768-4c60-8328-abe5665107e0" containerID="57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d" exitCode=0 Nov 01 22:22:46.123129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.122977 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" event={"ID":"2fae7803-f768-4c60-8328-abe5665107e0","Type":"ContainerDied","Data":"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d"} Nov 01 22:22:46.123129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.123038 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" event={"ID":"2fae7803-f768-4c60-8328-abe5665107e0","Type":"ContainerDied","Data":"d541124a42526179cf8094e314d23bb381b8b236430a3c0493f70ecf84c885cb"} Nov 01 22:22:46.123129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.123068 5036 scope.go:122] "RemoveContainer" containerID="57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d" Nov 01 22:22:46.123371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.123346 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" Nov 01 22:22:46.138758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.138738 5036 scope.go:122] "RemoveContainer" containerID="57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d" Nov 01 22:22:46.139013 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:46.138996 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d\": container with ID starting with 57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d not found: ID does not exist" containerID="57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d" Nov 01 22:22:46.139071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.139018 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d"} err="failed to get container status \"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d\": rpc error: code = NotFound desc = could not find container \"57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d\": container with ID starting with 57809e9a4d10b55a5611fea65c39fdcdf806b3bc27e10de4cbb4bbe95e2fe75d not found: ID does not exist" Nov 01 22:22:46.160436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.160414 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:22:46.165057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.165035 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb"] Nov 01 22:22:46.419708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.419624 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:46.488490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.486986 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fae7803-f768-4c60-8328-abe5665107e0" path="/var/lib/kubelet/pods/2fae7803-f768-4c60-8328-abe5665107e0/volumes" Nov 01 22:22:46.529917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:46.529864 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-da34ca/auth-test-model-da34ca-kserve-578f7b449c-4gprb" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:22:47.095746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:47.095707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:47.946388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:47.943316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:48.726484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:48.725002 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:48.859928 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:48.859893 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:22:49.467002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:49.466868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:49.508677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:49.508636 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:22:49.634469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:49.614648 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:50.127252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:50.127202 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:50.352639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:50.352600 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:50.561295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:50.561258 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:50.893515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:50.892051 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:51.314051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:51.314009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:52.033001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.032966 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:22:52.034148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034119 5036 cpu_manager.go:414] "containerMap: removing container" podUID="24eccbe5-cd28-4979-b9aa-2e2112acc2a4" containerName="revoke-keys" Nov 01 22:22:52.034148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034145 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24eccbe5-cd28-4979-b9aa-2e2112acc2a4" containerName="revoke-keys" Nov 01 22:22:52.034357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034191 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" Nov 01 22:22:52.034357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034201 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" Nov 01 22:22:52.034357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034274 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" Nov 01 22:22:52.034357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034283 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" Nov 01 22:22:52.034595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034462 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9532214b-a852-4158-8d4b-6543bd7cac1d" containerName="main" Nov 01 22:22:52.034595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034491 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fae7803-f768-4c60-8328-abe5665107e0" containerName="main" Nov 01 22:22:52.034595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.034508 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="24eccbe5-cd28-4979-b9aa-2e2112acc2a4" containerName="revoke-keys" Nov 01 22:22:52.044750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.044725 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.051269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.050515 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5a355b-serving-cert\"" Nov 01 22:22:52.051269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.050820 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:22:52.121787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.121754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:52.156385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.156349 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mhs\" (UniqueName: \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.156589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.156537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.156672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.156646 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.156797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.156778 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.252498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.252459 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:52.264710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.264673 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.268742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.265942 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mhs\" (UniqueName: \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.269389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.269357 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.269817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.269793 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.270532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.269573 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.270689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.270227 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.306520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.303311 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mhs\" (UniqueName: \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.329112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.329073 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.410824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.410776 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:52.930141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:52.930028 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:22:53.156892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.156788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:53.246413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.244067 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" event={"ID":"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c","Type":"ContainerStarted","Data":"eca8d1fb9fc64aa7aa8e75b916230eb4c490b719495d66b8451cde1365546b5c"} Nov 01 22:22:53.498925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.498785 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" containerID="cri-o://110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b" gracePeriod=45 Nov 01 22:22:53.503891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.503865 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:22:53.518926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.517707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:53.773487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.773288 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:53.895822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.895082 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" containerID="cri-o://8ce02d4d6f513510ca91fbbf9dd1f6f30968c078d2a742c33cf646f3465dfda5" gracePeriod=45 Nov 01 22:22:53.904809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.899714 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:22:53.990159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:53.990125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:54.023010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.022975 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023029 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023171 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023217 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023242 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023301 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\" (UniqueName: \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\") pod \"1179ed2c-c163-4c42-827b-2838746f0f43\" (UID: \"1179ed2c-c163-4c42-827b-2838746f0f43\") " Nov 01 22:22:54.023611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023425 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.023712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023691 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.023970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023952 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.024044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.023974 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.025097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.025056 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.028043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.027426 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.029596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.029574 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:54.032539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.032517 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6" pod "1179ed2c-c163-4c42-827b-2838746f0f43" (UID: "1179ed2c-c163-4c42-827b-2838746f0f43"). InnerVolumeSpecName "kube-api-access-tsfg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:54.137471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.128092 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.137471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.128130 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1179ed2c-c163-4c42-827b-2838746f0f43-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.137471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.128146 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfg6\" (UniqueName: \"kubernetes.io/projected/1179ed2c-c163-4c42-827b-2838746f0f43-kube-api-access-tsfg6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.137471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.128161 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1179ed2c-c163-4c42-827b-2838746f0f43-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.348128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.347882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" event={"ID":"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c","Type":"ContainerStarted","Data":"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065"} Nov 01 22:22:54.348128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.348021 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:22:54.351050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.350834 5036 generic.go:356] "Generic (PLEG): container finished" podID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerID="8ce02d4d6f513510ca91fbbf9dd1f6f30968c078d2a742c33cf646f3465dfda5" exitCode=0 Nov 01 22:22:54.351050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.351005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" event={"ID":"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5","Type":"ContainerDied","Data":"8ce02d4d6f513510ca91fbbf9dd1f6f30968c078d2a742c33cf646f3465dfda5"} Nov 01 22:22:54.354412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.353148 5036 generic.go:356] "Generic (PLEG): container finished" podID="1179ed2c-c163-4c42-827b-2838746f0f43" containerID="110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b" exitCode=0 Nov 01 22:22:54.354412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.353244 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" event={"ID":"1179ed2c-c163-4c42-827b-2838746f0f43","Type":"ContainerDied","Data":"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b"} Nov 01 22:22:54.354412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.353262 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" event={"ID":"1179ed2c-c163-4c42-827b-2838746f0f43","Type":"ContainerDied","Data":"82ee40b1c1b4db45611bac27e1fd032d3a69c2eb2d8f67cd373c8e67d25c111b"} Nov 01 22:22:54.354412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.353278 5036 scope.go:122] "RemoveContainer" containerID="110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b" Nov 01 22:22:54.354412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.353412 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb" Nov 01 22:22:54.377962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.377702 5036 scope.go:122] "RemoveContainer" containerID="110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b" Nov 01 22:22:54.379472 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:54.379383 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b\": container with ID starting with 110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b not found: ID does not exist" containerID="110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b" Nov 01 22:22:54.379472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.379417 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b"} err="failed to get container status \"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b\": rpc error: code = NotFound desc = could not find container \"110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b\": container with ID starting with 110866b28295d25c78c79715e937f1145f7fbf6baefe877b1398bbc84f348f8b not found: ID does not exist" Nov 01 22:22:54.384380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.384337 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" podStartSLOduration=1.747225201 podStartE2EDuration="2.384323835s" totalImagesPullingTime="637.098634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:54.376241477 +0000 UTC m=+2328.069221963" watchObservedRunningTime="2026-11-01 22:22:54.384323835 +0000 UTC m=+2328.077304317" Nov 01 22:22:54.417885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.417838 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:22:54.425635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.425527 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-702002/auth-test-model-702002-kserve-5f969cdd5c-9sdhb"] Nov 01 22:22:54.437912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.437886 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:22:54.446797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.446286 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" path="/var/lib/kubelet/pods/1179ed2c-c163-4c42-827b-2838746f0f43/volumes" Nov 01 22:22:54.527382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.527574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527506 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.527643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.527736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527720 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\" (UniqueName: \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.527788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.527830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.527815 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\") pod \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\" (UID: \"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5\") " Nov 01 22:22:54.529340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.528717 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.529678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.529523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.529803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.529781 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.533437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.532313 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "kube-api-access-9ph6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:54.533437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.532548 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:54.534342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.534200 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm" pod "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" (UID: "1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:54.543206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.543139 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.650991 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.651036 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.651053 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ph6r\" (UniqueName: \"kubernetes.io/projected/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-kube-api-access-9ph6r\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.651068 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.651092 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.652536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.651105 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:54.853477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.853407 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:54.996555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:54.992105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:55.420513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.409836 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" event={"ID":"1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5","Type":"ContainerDied","Data":"a35f8e27a93b089d13d38660bc722802ea6fbbeafdb6ef6cf3c9f6e1eef3ace0"} Nov 01 22:22:55.420513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.409899 5036 scope.go:122] "RemoveContainer" containerID="8ce02d4d6f513510ca91fbbf9dd1f6f30968c078d2a742c33cf646f3465dfda5" Nov 01 22:22:55.420513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.410027 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6" Nov 01 22:22:55.639469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.634499 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:22:55.679550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.674596 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/auth-test-model-e2bf63-kserve-775f7f777-ggvc6"] Nov 01 22:22:55.910127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:55.910078 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:56.075469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:56.074485 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" containerID="cri-o://d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950" gracePeriod=45 Nov 01 22:22:56.622986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:56.622946 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": dial tcp 10.132.1.12:8000: connect: connection refused" Nov 01 22:22:56.645682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:56.644596 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" path="/var/lib/kubelet/pods/1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5/volumes" Nov 01 22:22:56.876636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:56.876570 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:22:57.012954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.012918 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.012986 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013015 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\" (UniqueName: \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013040 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013143 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013191 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\") pod \"28fca3fe-885b-4442-aed8-be0edba2a2b2\" (UID: \"28fca3fe-885b-4442-aed8-be0edba2a2b2\") " Nov 01 22:22:57.013516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013426 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:57.013603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:57.013665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.013637 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:57.014051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.014019 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.014051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.014041 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.014051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.014055 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.015762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.015721 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "kube-api-access-cqtft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:57.015941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.015905 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:22:57.015993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.015960 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs" pod "28fca3fe-885b-4442-aed8-be0edba2a2b2" (UID: "28fca3fe-885b-4442-aed8-be0edba2a2b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:22:57.025517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.025458 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:57.115295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.115262 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fca3fe-885b-4442-aed8-be0edba2a2b2-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.115295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.115290 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fca3fe-885b-4442-aed8-be0edba2a2b2-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.115538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.115306 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqtft\" (UniqueName: \"kubernetes.io/projected/28fca3fe-885b-4442-aed8-be0edba2a2b2-kube-api-access-cqtft\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:57.654513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.654425 5036 generic.go:356] "Generic (PLEG): container finished" podID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerID="d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950" exitCode=0 Nov 01 22:22:57.654995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.654526 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" Nov 01 22:22:57.654995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.654522 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" event={"ID":"28fca3fe-885b-4442-aed8-be0edba2a2b2","Type":"ContainerDied","Data":"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950"} Nov 01 22:22:57.654995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.654648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8" event={"ID":"28fca3fe-885b-4442-aed8-be0edba2a2b2","Type":"ContainerDied","Data":"93e2a5e44290e952632b9774ae9104bc9e2ac9d4df1f6a6bd41dd5ab765f180e"} Nov 01 22:22:57.654995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.654665 5036 scope.go:122] "RemoveContainer" containerID="d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950" Nov 01 22:22:57.668093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.668077 5036 scope.go:122] "RemoveContainer" containerID="d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950" Nov 01 22:22:57.668390 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:22:57.668368 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950\": container with ID starting with d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950 not found: ID does not exist" containerID="d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950" Nov 01 22:22:57.668496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.668398 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950"} err="failed to get container status \"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950\": rpc error: code = NotFound desc = could not find container \"d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950\": container with ID starting with d04770f83953dbf889584afe4267be36ba8b9017ffb100d5cb14d91db1b84950 not found: ID does not exist" Nov 01 22:22:57.687930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.687898 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:22:57.695422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.695396 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e7e2df/test-model-e7e2df-kserve-5ddd9df877-xbzn8"] Nov 01 22:22:57.697184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:57.697157 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:58.440815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:58.440780 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" path="/var/lib/kubelet/pods/28fca3fe-885b-4442-aed8-be0edba2a2b2/volumes" Nov 01 22:22:58.445382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:58.445350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:59.225176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:59.225137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:59.932030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:22:59.931991 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:00.460831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:00.460799 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:23:00.493279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:00.493239 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:00.657747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:00.657706 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:01.395098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:01.395060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:02.165757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:02.165720 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:02.949953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:02.949916 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:03.740808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:03.740769 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:04.467937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.467894 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:04.572856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.572757 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" containerID="cri-o://55aa95234df2c001291f733f3baf3dad3e57ce2cc2b1fc15a1711b16a23374ee" gracePeriod=45 Nov 01 22:23:04.749137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.749058 5036 generic.go:356] "Generic (PLEG): container finished" podID="678519b4-d469-466d-93c7-8a668a4c2b30" containerID="55aa95234df2c001291f733f3baf3dad3e57ce2cc2b1fc15a1711b16a23374ee" exitCode=0 Nov 01 22:23:04.749583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.749133 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" event={"ID":"678519b4-d469-466d-93c7-8a668a4c2b30","Type":"ContainerDied","Data":"55aa95234df2c001291f733f3baf3dad3e57ce2cc2b1fc15a1711b16a23374ee"} Nov 01 22:23:04.923038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.923005 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:23:04.986198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.986469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.986469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986324 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\" (UniqueName: \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.986622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:04.986825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:04.986922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.986908 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.987199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.987181 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\" (UniqueName: \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.987199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.987182 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:04.987355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.987272 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\") pod \"678519b4-d469-466d-93c7-8a668a4c2b30\" (UID: \"678519b4-d469-466d-93c7-8a668a4c2b30\") " Nov 01 22:23:04.988198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.988174 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:04.988322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.988202 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:04.988322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.988217 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:04.989734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.989544 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "kube-api-access-ljmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:04.992224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.992188 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:04.992647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:04.992622 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm" pod "678519b4-d469-466d-93c7-8a668a4c2b30" (UID: "678519b4-d469-466d-93c7-8a668a4c2b30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:05.089597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.089554 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678519b4-d469-466d-93c7-8a668a4c2b30-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:05.089597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.089591 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678519b4-d469-466d-93c7-8a668a4c2b30-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:05.089597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.089603 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmjx\" (UniqueName: \"kubernetes.io/projected/678519b4-d469-466d-93c7-8a668a4c2b30-kube-api-access-ljmjx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:05.242782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.242739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:05.760400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.760370 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" Nov 01 22:23:05.760899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.760416 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg" event={"ID":"678519b4-d469-466d-93c7-8a668a4c2b30","Type":"ContainerDied","Data":"0af0fc619b03d2b18d52b21a1dc6cbd7c4040498db5717f4da026056f0a4a9dc"} Nov 01 22:23:05.760899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.760481 5036 scope.go:122] "RemoveContainer" containerID="55aa95234df2c001291f733f3baf3dad3e57ce2cc2b1fc15a1711b16a23374ee" Nov 01 22:23:05.787138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.787104 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:23:05.800035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.800004 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-de0cba/auth-test-model-de0cba-kserve-8456974769-g58dg"] Nov 01 22:23:05.942227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:05.942190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:06.038363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.038321 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:23:06.077378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.077322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:06.283705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.283643 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh"] Nov 01 22:23:06.285456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.285403 5036 cpu_manager.go:414] "containerMap: removing container" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" Nov 01 22:23:06.285738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.285719 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" Nov 01 22:23:06.285948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.285933 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" Nov 01 22:23:06.286061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286048 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" Nov 01 22:23:06.286173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286160 5036 cpu_manager.go:414] "containerMap: removing container" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" Nov 01 22:23:06.286256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286245 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" Nov 01 22:23:06.286417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286404 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" Nov 01 22:23:06.286591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286574 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" Nov 01 22:23:06.288204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.286881 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" containerName="main" Nov 01 22:23:06.288411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.288362 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1179ed2c-c163-4c42-827b-2838746f0f43" containerName="main" Nov 01 22:23:06.288543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.288531 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fca3fe-885b-4442-aed8-be0edba2a2b2" containerName="main" Nov 01 22:23:06.288639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.288628 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd4129f-b47a-40c7-ac51-7aee9d2d1bc5" containerName="main" Nov 01 22:23:06.307469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.306952 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh"] Nov 01 22:23:06.307469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.307078 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.316711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.316682 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:23:06.319438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.316952 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:23:06.418264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.418228 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.418463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.418376 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8f4t\" (UniqueName: \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.418537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.418516 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.446933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.446896 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678519b4-d469-466d-93c7-8a668a4c2b30" path="/var/lib/kubelet/pods/678519b4-d469-466d-93c7-8a668a4c2b30/volumes" Nov 01 22:23:06.520004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.519971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8f4t\" (UniqueName: \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.520184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.520130 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.520912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.520606 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.521054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.521009 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.523247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.523221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.542507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.534693 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8f4t\" (UniqueName: \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\") pod \"maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.661707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.661677 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:23:06.668714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.668687 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:06.959160 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:06.959132 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be3d173_8157_40a2_abfe_ac6125b6c496.slice/crio-19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714 WatchSource:0}: Error finding container 19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714: Status 404 returned error can't find the container with id 19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714 Nov 01 22:23:06.959601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.959439 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh"] Nov 01 22:23:06.998001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:06.997970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:07.742905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:07.742821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:07.852403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:07.852365 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" event={"ID":"6be3d173-8157-40a2-abfe-ac6125b6c496","Type":"ContainerStarted","Data":"fa52deab85a4dda2b3b1107b9a3cb945720f885c03c35894483cba32709cb93b"} Nov 01 22:23:07.852597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:07.852419 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" event={"ID":"6be3d173-8157-40a2-abfe-ac6125b6c496","Type":"ContainerStarted","Data":"19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714"} Nov 01 22:23:07.872354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:07.872311 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" podStartSLOduration=1.872297318 podStartE2EDuration="1.872297318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.870338512 +0000 UTC m=+2341.563318998" watchObservedRunningTime="2026-11-01 22:23:07.872297318 +0000 UTC m=+2341.565277823" Nov 01 22:23:08.504384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:08.504289 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:08.863487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:08.863436 5036 generic.go:356] "Generic (PLEG): container finished" podID="6be3d173-8157-40a2-abfe-ac6125b6c496" containerID="fa52deab85a4dda2b3b1107b9a3cb945720f885c03c35894483cba32709cb93b" exitCode=0 Nov 01 22:23:08.863487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:08.863476 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" event={"ID":"6be3d173-8157-40a2-abfe-ac6125b6c496","Type":"ContainerDied","Data":"fa52deab85a4dda2b3b1107b9a3cb945720f885c03c35894483cba32709cb93b"} Nov 01 22:23:09.239538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:09.235190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:10.024773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:09.991175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:10.111718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.111678 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:10.121872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.121843 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\") pod \"6be3d173-8157-40a2-abfe-ac6125b6c496\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " Nov 01 22:23:10.122060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.121941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\") pod \"6be3d173-8157-40a2-abfe-ac6125b6c496\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " Nov 01 22:23:10.122060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.121987 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\" (UniqueName: \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\") pod \"6be3d173-8157-40a2-abfe-ac6125b6c496\" (UID: \"6be3d173-8157-40a2-abfe-ac6125b6c496\") " Nov 01 22:23:10.122813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.122784 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca" pod "6be3d173-8157-40a2-abfe-ac6125b6c496" (UID: "6be3d173-8157-40a2-abfe-ac6125b6c496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:10.124664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.124471 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t" pod "6be3d173-8157-40a2-abfe-ac6125b6c496" (UID: "6be3d173-8157-40a2-abfe-ac6125b6c496"). InnerVolumeSpecName "kube-api-access-b8f4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:10.125849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.125822 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca" pod "6be3d173-8157-40a2-abfe-ac6125b6c496" (UID: "6be3d173-8157-40a2-abfe-ac6125b6c496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:10.224081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.224044 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:10.224081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.224081 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3d173-8157-40a2-abfe-ac6125b6c496-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:10.224340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.224093 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8f4t\" (UniqueName: \"kubernetes.io/projected/6be3d173-8157-40a2-abfe-ac6125b6c496-kube-api-access-b8f4t\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:10.490150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.490118 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:10.615502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.615465 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:10.747166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.747044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:10.883272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.883235 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" Nov 01 22:23:10.883272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.883246 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh" event={"ID":"6be3d173-8157-40a2-abfe-ac6125b6c496","Type":"ContainerDied","Data":"19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714"} Nov 01 22:23:10.883587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:10.883304 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19849fceeae4ea68e9e72cfc4f0ffa3ad74469de72a760634b247e06cffc6714" Nov 01 22:23:11.561009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:11.560935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:12.223551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.223506 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:23:12.263470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.225066 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6be3d173-8157-40a2-abfe-ac6125b6c496" containerName="revoke-keys" Nov 01 22:23:12.306412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.274465 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be3d173-8157-40a2-abfe-ac6125b6c496" containerName="revoke-keys" Nov 01 22:23:12.306412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.274814 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be3d173-8157-40a2-abfe-ac6125b6c496" containerName="revoke-keys" Nov 01 22:23:12.317464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.311759 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:23:12.317464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.311895 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.362469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.358063 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e25317-serving-cert\"" Nov 01 22:23:12.458749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.458653 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:12.518344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.515240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmk6\" (UniqueName: \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.518344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.515375 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.518344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.515639 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.518344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.515724 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.635350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.635312 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:12.637554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.637500 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.637977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.637951 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmk6\" (UniqueName: \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.639204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.639180 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.639673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.639652 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.642774 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:23:12.640061 5036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-serving-cert: secret "maas-api-e2e-ipp-a-e25317-serving-cert" not found Nov 01 22:23:12.642970 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:23:12.642955 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls podName:8a8d4eb0-7f6a-4827-9178-176f2cd3eae4 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.142932367 +0000 UTC m=+2346.835912846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls") pod "maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" (UID: "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4") : secret "maas-api-e2e-ipp-a-e25317-serving-cert" not found Nov 01 22:23:12.643327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.640366 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.643482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.643324 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.652244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.652221 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmk6\" (UniqueName: \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:12.777086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:12.777047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:13.153523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.151466 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:13.194467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.185053 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:13.425083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.424984 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:13.698980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.698888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:13.793538 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:13.793502 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8d4eb0_7f6a_4827_9178_176f2cd3eae4.slice/crio-a5ee1fcafce336ad797b75066dfe38c5f00f7bbb2d42bdd3f2eb345f34a8b9f0 WatchSource:0}: Error finding container a5ee1fcafce336ad797b75066dfe38c5f00f7bbb2d42bdd3f2eb345f34a8b9f0: Status 404 returned error can't find the container with id a5ee1fcafce336ad797b75066dfe38c5f00f7bbb2d42bdd3f2eb345f34a8b9f0 Nov 01 22:23:13.796148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.796115 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:23:13.916120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:13.916076 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" event={"ID":"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4","Type":"ContainerStarted","Data":"a5ee1fcafce336ad797b75066dfe38c5f00f7bbb2d42bdd3f2eb345f34a8b9f0"} Nov 01 22:23:14.540865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:14.540761 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:14.929064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:14.928968 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" event={"ID":"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4","Type":"ContainerStarted","Data":"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858"} Nov 01 22:23:14.929514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:14.929216 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:14.951213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:14.951157 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" podStartSLOduration=2.576906235 podStartE2EDuration="2.951142022s" totalImagesPullingTime="374.235787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:14.949111632 +0000 UTC m=+2348.642092119" watchObservedRunningTime="2026-11-01 22:23:14.951142022 +0000 UTC m=+2348.644122508" Nov 01 22:23:15.251263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:15.251173 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:15.383300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:15.383261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:15.517070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:15.516981 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:16.218635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:16.218598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:16.540464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:16.537742 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:16.691476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:16.688424 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:17.138894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:17.138857 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:17.224729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:17.224688 5036 scope.go:122] "RemoveContainer" containerID="706382ae5fa4bc6fc8126c39d4dad9200170569762094b6b09d6c05cc38145ff" Nov 01 22:23:17.237573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:17.237541 5036 scope.go:122] "RemoveContainer" containerID="e32e5c3a26d67e18b514fc7c2d7194d70cacdf50887113ab955aea3043ac9ae6" Nov 01 22:23:17.247991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:17.247964 5036 scope.go:122] "RemoveContainer" containerID="9a1f40a36992511b25c3958d608e6cca07b67f57569b5a299b8dfaef02631570" Nov 01 22:23:17.898729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:17.898685 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.535511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:18.535432 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.832863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:18.832771 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.866743 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:23:18.866709 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:23:19.516029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:19.515994 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:20.227597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:20.227553 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:20.957047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:20.957014 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:23:21.040792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.040751 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:21.077985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.077944 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" containerID="cri-o://759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7" gracePeriod=45 Nov 01 22:23:21.387020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.386990 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:23:21.468300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468262 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\" (UniqueName: \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.468532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468508 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.468625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.468774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468620 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.468774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468751 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.468898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.468785 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\") pod \"7ff32747-47d4-4e81-8606-1eec5d959069\" (UID: \"7ff32747-47d4-4e81-8606-1eec5d959069\") " Nov 01 22:23:21.472154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.471369 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:21.472154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.471599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:21.472154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.471668 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:21.472849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.472814 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:21.472985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.472918 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:21.472985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.472868 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz" pod "7ff32747-47d4-4e81-8606-1eec5d959069" (UID: "7ff32747-47d4-4e81-8606-1eec5d959069"). InnerVolumeSpecName "kube-api-access-tbgpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:21.570543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570506 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.570543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570537 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.570738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570550 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.570738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570562 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff32747-47d4-4e81-8606-1eec5d959069-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.570738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570575 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff32747-47d4-4e81-8606-1eec5d959069-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.570738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.570587 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbgpz\" (UniqueName: \"kubernetes.io/projected/7ff32747-47d4-4e81-8606-1eec5d959069-kube-api-access-tbgpz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.818748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:21.818709 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.085260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.085217 5036 generic.go:356] "Generic (PLEG): container finished" podID="7ff32747-47d4-4e81-8606-1eec5d959069" containerID="759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7" exitCode=0 Nov 01 22:23:22.085260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.085264 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" event={"ID":"7ff32747-47d4-4e81-8606-1eec5d959069","Type":"ContainerDied","Data":"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7"} Nov 01 22:23:22.085593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.085297 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" Nov 01 22:23:22.085593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.085316 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt" event={"ID":"7ff32747-47d4-4e81-8606-1eec5d959069","Type":"ContainerDied","Data":"422914fc11e1bd67b9f0ecfd51d9083ca91672d2d9d769ed75746bcc5d8bfaf7"} Nov 01 22:23:22.085593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.085340 5036 scope.go:122] "RemoveContainer" containerID="759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7" Nov 01 22:23:22.099749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.099731 5036 scope.go:122] "RemoveContainer" containerID="759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7" Nov 01 22:23:22.100032 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:23:22.100014 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7\": container with ID starting with 759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7 not found: ID does not exist" containerID="759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7" Nov 01 22:23:22.100082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.100040 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7"} err="failed to get container status \"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7\": rpc error: code = NotFound desc = could not find container \"759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7\": container with ID starting with 759d58a0f222b77f3fce9569541ebb841dd458e6b884c6475d1572cff19eabf7 not found: ID does not exist" Nov 01 22:23:22.116828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.116796 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:23:22.121757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.121728 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6b4c49/test-model-6b4c49-kserve-6594bc78f5-74mkt"] Nov 01 22:23:22.441003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.440926 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" path="/var/lib/kubelet/pods/7ff32747-47d4-4e81-8606-1eec5d959069/volumes" Nov 01 22:23:22.565234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:22.565197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:23.217086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:23.217039 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:23.943840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:23.943797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:24.642626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:24.642582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:25.287801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:25.287760 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.121493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.114880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.671473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.671369 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.828487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.826620 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh"] Nov 01 22:23:26.829465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.829287 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.888222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.888189 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e7e2df-xwrhh"] Nov 01 22:23:26.971604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:26.971497 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.268649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:27.268614 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.969693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:27.969657 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.523941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:28.523899 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be3d173-8157-40a2-abfe-ac6125b6c496" path="/var/lib/kubelet/pods/6be3d173-8157-40a2-abfe-ac6125b6c496/volumes" Nov 01 22:23:29.379846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:29.379804 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:30.296895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:30.296852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.137628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:31.137591 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.876702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:31.876661 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:32.286686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.286653 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn"] Nov 01 22:23:32.287417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.287399 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" Nov 01 22:23:32.287481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.287419 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" Nov 01 22:23:32.287592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.287580 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff32747-47d4-4e81-8606-1eec5d959069" containerName="main" Nov 01 22:23:32.291246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.291227 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.296487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.295369 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:23:32.296487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.295516 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:23:32.299708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.299683 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn"] Nov 01 22:23:32.419229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.418235 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.419949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.419914 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.420074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.420038 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcb5\" (UniqueName: \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.485952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.485916 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:32.521368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.521333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.521569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.521429 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcb5\" (UniqueName: \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.521569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.521549 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.522114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.522081 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.523912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.523889 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.537377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.537306 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcb5\" (UniqueName: \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\") pod \"maas-api-revoke-sub-2b609be6-dwddn\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.620377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.620341 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:32.643413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:32.643322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.170108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.170078 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn"] Nov 01 22:23:33.170267 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:33.170218 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d40012f_4004_438d_bec0_020f6544dc96.slice/crio-690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf WatchSource:0}: Error finding container 690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf: Status 404 returned error can't find the container with id 690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf Nov 01 22:23:33.191852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.191821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.709961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.709919 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.810105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.810066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" event={"ID":"3d40012f-4004-438d-bec0-020f6544dc96","Type":"ContainerStarted","Data":"7200acd54af5075b451f8281dbfb167c0ad39e35d1754d7170af35018d59f9ed"} Nov 01 22:23:33.810325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.810152 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" event={"ID":"3d40012f-4004-438d-bec0-020f6544dc96","Type":"ContainerStarted","Data":"690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf"} Nov 01 22:23:33.831231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.831185 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" podStartSLOduration=1.831172929 podStartE2EDuration="1.831172929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:33.828890774 +0000 UTC m=+2367.521871260" watchObservedRunningTime="2026-11-01 22:23:33.831172929 +0000 UTC m=+2367.524153415" Nov 01 22:23:33.843339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:33.843263 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:34.213390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.197489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:34.448083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.448047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:34.597132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.597094 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg"] Nov 01 22:23:34.602127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.602059 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.611522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.611490 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg"] Nov 01 22:23:34.627496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.627461 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.627772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.627746 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b4p\" (UniqueName: \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.628089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.628047 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.729435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.729399 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.729942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.729536 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.729942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.729612 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b4p\" (UniqueName: \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.730089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.730066 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.732195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.732166 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.743103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.743072 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b4p\" (UniqueName: \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\") pod \"maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:34.821497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.821427 5036 generic.go:356] "Generic (PLEG): container finished" podID="3d40012f-4004-438d-bec0-020f6544dc96" containerID="7200acd54af5075b451f8281dbfb167c0ad39e35d1754d7170af35018d59f9ed" exitCode=0 Nov 01 22:23:34.821497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.821473 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" event={"ID":"3d40012f-4004-438d-bec0-020f6544dc96","Type":"ContainerDied","Data":"7200acd54af5075b451f8281dbfb167c0ad39e35d1754d7170af35018d59f9ed"} Nov 01 22:23:34.924633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:34.924547 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:35.210927 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:35.210891 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32391d7_06e6_419d_9bc5_1d24615d9969.slice/crio-b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1 WatchSource:0}: Error finding container b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1: Status 404 returned error can't find the container with id b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1 Nov 01 22:23:35.213552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:35.213523 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg"] Nov 01 22:23:35.643860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:35.643823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.834106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:35.834073 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" event={"ID":"d32391d7-06e6-419d-9bc5-1d24615d9969","Type":"ContainerStarted","Data":"8cfc6b1089b27f84b563df0b44511cc0ce2a856964bcb5a5222c96cd910faea3"} Nov 01 22:23:35.834636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:35.834125 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" event={"ID":"d32391d7-06e6-419d-9bc5-1d24615d9969","Type":"ContainerStarted","Data":"b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1"} Nov 01 22:23:35.857372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:35.857283 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" podStartSLOduration=1.8572678489999999 podStartE2EDuration="1.857267849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:35.852438859 +0000 UTC m=+2369.545419344" watchObservedRunningTime="2026-11-01 22:23:35.857267849 +0000 UTC m=+2369.550248335" Nov 01 22:23:36.012404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.012379 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:36.049965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.049919 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\") pod \"3d40012f-4004-438d-bec0-020f6544dc96\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " Nov 01 22:23:36.050134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.049984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\" (UniqueName: \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\") pod \"3d40012f-4004-438d-bec0-020f6544dc96\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " Nov 01 22:23:36.050134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.050084 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\") pod \"3d40012f-4004-438d-bec0-020f6544dc96\" (UID: \"3d40012f-4004-438d-bec0-020f6544dc96\") " Nov 01 22:23:36.050601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.050568 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca" pod "3d40012f-4004-438d-bec0-020f6544dc96" (UID: "3d40012f-4004-438d-bec0-020f6544dc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:36.051079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.051051 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:36.053166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.053127 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca" pod "3d40012f-4004-438d-bec0-020f6544dc96" (UID: "3d40012f-4004-438d-bec0-020f6544dc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:36.053621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.053599 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5" pod "3d40012f-4004-438d-bec0-020f6544dc96" (UID: "3d40012f-4004-438d-bec0-020f6544dc96"). InnerVolumeSpecName "kube-api-access-nbcb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:36.152829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.152749 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcb5\" (UniqueName: \"kubernetes.io/projected/3d40012f-4004-438d-bec0-020f6544dc96-kube-api-access-nbcb5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:36.152829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.152781 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d40012f-4004-438d-bec0-020f6544dc96-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:36.491134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.491095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:36.854589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.854550 5036 generic.go:356] "Generic (PLEG): container finished" podID="d32391d7-06e6-419d-9bc5-1d24615d9969" containerID="8cfc6b1089b27f84b563df0b44511cc0ce2a856964bcb5a5222c96cd910faea3" exitCode=0 Nov 01 22:23:36.855005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.854628 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" event={"ID":"d32391d7-06e6-419d-9bc5-1d24615d9969","Type":"ContainerDied","Data":"8cfc6b1089b27f84b563df0b44511cc0ce2a856964bcb5a5222c96cd910faea3"} Nov 01 22:23:36.856472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.856453 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" Nov 01 22:23:36.856631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.856472 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn" event={"ID":"3d40012f-4004-438d-bec0-020f6544dc96","Type":"ContainerDied","Data":"690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf"} Nov 01 22:23:36.856631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.856517 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690557dce798b550f90102ac95bccc7b6a89a48ab00a824042e607a87ddddabf" Nov 01 22:23:36.870038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.870008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:36.988866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:36.988825 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:37.420879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:37.420841 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:38.047949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.047923 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:38.095882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.095823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\" (UniqueName: \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\") pod \"d32391d7-06e6-419d-9bc5-1d24615d9969\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " Nov 01 22:23:38.096591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.096562 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\") pod \"d32391d7-06e6-419d-9bc5-1d24615d9969\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " Nov 01 22:23:38.096720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.096667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\") pod \"d32391d7-06e6-419d-9bc5-1d24615d9969\" (UID: \"d32391d7-06e6-419d-9bc5-1d24615d9969\") " Nov 01 22:23:38.097205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.097182 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca" pod "d32391d7-06e6-419d-9bc5-1d24615d9969" (UID: "d32391d7-06e6-419d-9bc5-1d24615d9969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:38.099099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.099068 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p" pod "d32391d7-06e6-419d-9bc5-1d24615d9969" (UID: "d32391d7-06e6-419d-9bc5-1d24615d9969"). InnerVolumeSpecName "kube-api-access-28b4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:38.100646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.100623 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca" pod "d32391d7-06e6-419d-9bc5-1d24615d9969" (UID: "d32391d7-06e6-419d-9bc5-1d24615d9969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:38.148619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.148542 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:38.199978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.199936 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:38.199978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.199978 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32391d7-06e6-419d-9bc5-1d24615d9969-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:38.199978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.199994 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28b4p\" (UniqueName: \"kubernetes.io/projected/d32391d7-06e6-419d-9bc5-1d24615d9969-kube-api-access-28b4p\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:38.877158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.877121 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" Nov 01 22:23:38.877158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.877147 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg" event={"ID":"d32391d7-06e6-419d-9bc5-1d24615d9969","Type":"ContainerDied","Data":"b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1"} Nov 01 22:23:38.877397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.877210 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24706615764041e7f566b13ee9e802b271a969d97adda6bfbacaac89eb584d1" Nov 01 22:23:38.949346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:38.949306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:39.228594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:39.228047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:39.356643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:39.356606 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:39.489330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:39.489233 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:39.946986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:39.946946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:40.921884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:40.921844 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:41.738087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:41.738052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:42.419148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.419104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:42.507102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.506968 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m"] Nov 01 22:23:42.508105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508076 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d32391d7-06e6-419d-9bc5-1d24615d9969" containerName="revoke-keys" Nov 01 22:23:42.508105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508104 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32391d7-06e6-419d-9bc5-1d24615d9969" containerName="revoke-keys" Nov 01 22:23:42.508296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508156 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3d40012f-4004-438d-bec0-020f6544dc96" containerName="revoke-keys" Nov 01 22:23:42.508296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508164 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d40012f-4004-438d-bec0-020f6544dc96" containerName="revoke-keys" Nov 01 22:23:42.508404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508337 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32391d7-06e6-419d-9bc5-1d24615d9969" containerName="revoke-keys" Nov 01 22:23:42.508404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.508361 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d40012f-4004-438d-bec0-020f6544dc96" containerName="revoke-keys" Nov 01 22:23:42.512869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.512848 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.521415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.521389 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:23:42.521601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.521579 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:23:42.528184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.527039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m"] Nov 01 22:23:42.597555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.597011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.597555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.597122 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.597555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.597326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjs8r\" (UniqueName: \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.699167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.699078 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjs8r\" (UniqueName: \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.699308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.699238 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.699365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.699314 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.700013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.699982 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.701698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.701676 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.711779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.711759 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjs8r\" (UniqueName: \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\") pod \"maas-api-revoke-sub-374ff4f5-wdm2m\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:42.845019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:42.844988 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:43.133887 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:43.133854 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac11d2b2_4a0f_4849_989a_6b737d0bc156.slice/crio-e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353 WatchSource:0}: Error finding container e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353: Status 404 returned error can't find the container with id e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353 Nov 01 22:23:43.135884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.135832 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m"] Nov 01 22:23:43.176562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.176533 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:43.851809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.851777 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:43.950483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.950064 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" event={"ID":"ac11d2b2-4a0f-4849-989a-6b737d0bc156","Type":"ContainerStarted","Data":"e4709177a2355ffe8f6f3d7bab964ab6228060d3f0e65402845b4b74b78fad4f"} Nov 01 22:23:43.950483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.950133 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" event={"ID":"ac11d2b2-4a0f-4849-989a-6b737d0bc156","Type":"ContainerStarted","Data":"e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353"} Nov 01 22:23:43.980993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:43.980547 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" podStartSLOduration=1.9805209339999998 podStartE2EDuration="1.980520934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:43.976719813 +0000 UTC m=+2377.669700456" watchObservedRunningTime="2026-11-01 22:23:43.980520934 +0000 UTC m=+2377.673501422" Nov 01 22:23:44.638392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:44.638351 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:44.873838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:44.873800 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:44.959681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:44.959587 5036 generic.go:356] "Generic (PLEG): container finished" podID="ac11d2b2-4a0f-4849-989a-6b737d0bc156" containerID="e4709177a2355ffe8f6f3d7bab964ab6228060d3f0e65402845b4b74b78fad4f" exitCode=0 Nov 01 22:23:44.959845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:44.959677 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" event={"ID":"ac11d2b2-4a0f-4849-989a-6b737d0bc156","Type":"ContainerDied","Data":"e4709177a2355ffe8f6f3d7bab964ab6228060d3f0e65402845b4b74b78fad4f"} Nov 01 22:23:45.026924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:45.026848 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:45.520932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:45.520888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.149586 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.149552 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:46.289569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.289529 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\") pod \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " Nov 01 22:23:46.289766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.289601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\") pod \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " Nov 01 22:23:46.289766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.289708 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\" (UniqueName: \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\") pod \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\" (UID: \"ac11d2b2-4a0f-4849-989a-6b737d0bc156\") " Nov 01 22:23:46.289982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.289951 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca" pod "ac11d2b2-4a0f-4849-989a-6b737d0bc156" (UID: "ac11d2b2-4a0f-4849-989a-6b737d0bc156"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:46.290491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.290437 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:46.292039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.292017 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca" pod "ac11d2b2-4a0f-4849-989a-6b737d0bc156" (UID: "ac11d2b2-4a0f-4849-989a-6b737d0bc156"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:46.292475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.292456 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r" pod "ac11d2b2-4a0f-4849-989a-6b737d0bc156" (UID: "ac11d2b2-4a0f-4849-989a-6b737d0bc156"). InnerVolumeSpecName "kube-api-access-hjs8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:46.341665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.341632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.392374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.392300 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac11d2b2-4a0f-4849-989a-6b737d0bc156-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:46.392374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.392342 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjs8r\" (UniqueName: \"kubernetes.io/projected/ac11d2b2-4a0f-4849-989a-6b737d0bc156-kube-api-access-hjs8r\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:46.977592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.977544 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" Nov 01 22:23:46.977592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.977583 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m" event={"ID":"ac11d2b2-4a0f-4849-989a-6b737d0bc156","Type":"ContainerDied","Data":"e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353"} Nov 01 22:23:46.978999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:46.977642 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e106d6955616d5a0fa8379247ad5d4ab54e0803067063864d9af96fb940ec353" Nov 01 22:23:47.061421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:47.061383 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:47.838138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:47.838028 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:48.625106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:48.625053 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:48.860603 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:23:48.860570 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:23:49.433393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:49.433354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:50.153948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:50.153905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:50.950768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:50.950727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:51.651652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:51.651612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:52.407206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:52.407164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:52.544614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:52.544579 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:52.832558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:52.832520 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:52.955734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:52.955695 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:53.099863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.099774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:53.294469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.293556 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4"] Nov 01 22:23:53.298467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.294737 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ac11d2b2-4a0f-4849-989a-6b737d0bc156" containerName="revoke-keys" Nov 01 22:23:53.298467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.294767 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac11d2b2-4a0f-4849-989a-6b737d0bc156" containerName="revoke-keys" Nov 01 22:23:53.298467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.294983 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac11d2b2-4a0f-4849-989a-6b737d0bc156" containerName="revoke-keys" Nov 01 22:23:53.341467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.341064 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.355517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.345954 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4"] Nov 01 22:23:53.367029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.366994 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:23:53.367999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.367947 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:23:53.448564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.448525 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5fv8\" (UniqueName: \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.448839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.448816 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.449120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.449103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.568014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.567964 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.568419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.568402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5fv8\" (UniqueName: \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.568595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.568582 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.574911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.572624 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.574911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.574699 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.625284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.625166 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5fv8\" (UniqueName: \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\") pod \"maas-api-revoke-sub-518b56ba-nk4z4\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.785300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.785264 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:53.839351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:53.839277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:54.149704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:54.149671 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4"] Nov 01 22:23:54.150592 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:23:54.150566 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed428a9_fcb6_4766_9547_030c1935c347.slice/crio-81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296 WatchSource:0}: Error finding container 81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296: Status 404 returned error can't find the container with id 81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296 Nov 01 22:23:54.628478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:54.626357 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.079662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.078190 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" event={"ID":"fed428a9-fcb6-4766-9547-030c1935c347","Type":"ContainerStarted","Data":"7ba925241730cba3a2c56d0975678c99fb0bb97227dca4f01463af15cd19f638"} Nov 01 22:23:55.079662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.078258 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" event={"ID":"fed428a9-fcb6-4766-9547-030c1935c347","Type":"ContainerStarted","Data":"81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296"} Nov 01 22:23:55.121504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.117984 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" podStartSLOduration=2.11795794 podStartE2EDuration="2.11795794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:55.096874239 +0000 UTC m=+2388.789854740" watchObservedRunningTime="2026-11-01 22:23:55.11795794 +0000 UTC m=+2388.810938428" Nov 01 22:23:55.252102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.252060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.416524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.409522 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.556497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.556459 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.697540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.685239 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg"] Nov 01 22:23:55.767469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.760223 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.876494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:55.875658 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b4c49-4j7zg"] Nov 01 22:23:56.027289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.027256 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv"] Nov 01 22:23:56.031858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.031831 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91ac00c-9mnrv"] Nov 01 22:23:56.089313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.089279 5036 generic.go:356] "Generic (PLEG): container finished" podID="fed428a9-fcb6-4766-9547-030c1935c347" containerID="7ba925241730cba3a2c56d0975678c99fb0bb97227dca4f01463af15cd19f638" exitCode=0 Nov 01 22:23:56.089313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.089312 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" event={"ID":"fed428a9-fcb6-4766-9547-030c1935c347","Type":"ContainerDied","Data":"7ba925241730cba3a2c56d0975678c99fb0bb97227dca4f01463af15cd19f638"} Nov 01 22:23:56.111909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.111875 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.440519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.440402 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32391d7-06e6-419d-9bc5-1d24615d9969" path="/var/lib/kubelet/pods/d32391d7-06e6-419d-9bc5-1d24615d9969/volumes" Nov 01 22:23:56.440976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.440954 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18a5f2b-3f20-4779-acf8-4ac85aaea68a" path="/var/lib/kubelet/pods/f18a5f2b-3f20-4779-acf8-4ac85aaea68a/volumes" Nov 01 22:23:56.622858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.622812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.845019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.844978 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.972007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:56.971956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:57.303301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.303268 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:57.419482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.419342 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\") pod \"fed428a9-fcb6-4766-9547-030c1935c347\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " Nov 01 22:23:57.419482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.419404 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\") pod \"fed428a9-fcb6-4766-9547-030c1935c347\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " Nov 01 22:23:57.419482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.419470 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\" (UniqueName: \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\") pod \"fed428a9-fcb6-4766-9547-030c1935c347\" (UID: \"fed428a9-fcb6-4766-9547-030c1935c347\") " Nov 01 22:23:57.419865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.419836 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca" pod "fed428a9-fcb6-4766-9547-030c1935c347" (UID: "fed428a9-fcb6-4766-9547-030c1935c347"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:57.420281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.420257 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:57.421941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.421908 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca" pod "fed428a9-fcb6-4766-9547-030c1935c347" (UID: "fed428a9-fcb6-4766-9547-030c1935c347"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:57.422094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.422050 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8" pod "fed428a9-fcb6-4766-9547-030c1935c347" (UID: "fed428a9-fcb6-4766-9547-030c1935c347"). InnerVolumeSpecName "kube-api-access-b5fv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:57.445771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.445736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:57.522139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.522106 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed428a9-fcb6-4766-9547-030c1935c347-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:57.522139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.522138 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5fv8\" (UniqueName: \"kubernetes.io/projected/fed428a9-fcb6-4766-9547-030c1935c347-kube-api-access-b5fv8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:57.946168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:57.946133 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:23:58.067324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:58.067292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.107997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:58.107967 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" event={"ID":"fed428a9-fcb6-4766-9547-030c1935c347","Type":"ContainerDied","Data":"81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296"} Nov 01 22:23:58.108178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:58.108014 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81cf813e6337bca11b4ff32a1cd1d55b084454a9e0aab5e9f22d74ad51644296" Nov 01 22:23:58.108178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:58.108014 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4" Nov 01 22:23:58.265530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:58.263652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:59.116847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:23:59.111019 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:00.426841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:00.426805 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:01.945312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:01.945270 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:02.104466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:02.101103 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:02.788731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:02.788676 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:02.937465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:02.931824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:03.065342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.065248 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:24:03.066210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.066169 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" containerID="cri-o://b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0" gracePeriod=30 Nov 01 22:24:03.134603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.134562 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:03.659836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.659800 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l"] Nov 01 22:24:03.660577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.660556 5036 cpu_manager.go:414] "containerMap: removing container" podUID="fed428a9-fcb6-4766-9547-030c1935c347" containerName="revoke-keys" Nov 01 22:24:03.660577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.660574 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed428a9-fcb6-4766-9547-030c1935c347" containerName="revoke-keys" Nov 01 22:24:03.660811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.660797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed428a9-fcb6-4766-9547-030c1935c347" containerName="revoke-keys" Nov 01 22:24:03.664984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.664963 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.673236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.673210 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l"] Nov 01 22:24:03.707670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.707640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.707845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.707787 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.708040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.708020 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xppd\" (UniqueName: \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.709692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.709668 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:24:03.710236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.710211 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:24:03.810319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.810280 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.810716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.810693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xppd\" (UniqueName: \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.810872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.810849 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.811119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.810953 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.813337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.813308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.825373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.825348 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xppd\" (UniqueName: \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\") pod \"maas-api-revoke-sub-66ca4a5e-jpw6l\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:03.842755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.842727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:03.989155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:03.989064 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:04.290479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:04.290432 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l"] Nov 01 22:24:04.291146 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:24:04.291118 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81975588_5e48_4d1b_b9b7_940bd0dfcee1.slice/crio-45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5 WatchSource:0}: Error finding container 45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5: Status 404 returned error can't find the container with id 45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5 Nov 01 22:24:04.343302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:04.343277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:04.355600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:04.355573 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" event={"ID":"81975588-5e48-4d1b-b9b7-940bd0dfcee1","Type":"ContainerStarted","Data":"45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5"} Nov 01 22:24:04.550668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:04.550583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:04.774468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:04.773865 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:05.256776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:05.256724 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:05.366917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:05.366877 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" event={"ID":"81975588-5e48-4d1b-b9b7-940bd0dfcee1","Type":"ContainerStarted","Data":"1cfc38831a58f45c88c298ad5db3ad2c39a376bafdbb5cc84a8f20c4d15b2a21"} Nov 01 22:24:05.389073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:05.389017 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" podStartSLOduration=2.389003729 podStartE2EDuration="2.389003729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:05.385994156 +0000 UTC m=+2399.078974644" watchObservedRunningTime="2026-11-01 22:24:05.389003729 +0000 UTC m=+2399.081984214" Nov 01 22:24:05.539858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:05.539820 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:06.228220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:06.228173 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:06.377312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:06.377273 5036 generic.go:356] "Generic (PLEG): container finished" podID="81975588-5e48-4d1b-b9b7-940bd0dfcee1" containerID="1cfc38831a58f45c88c298ad5db3ad2c39a376bafdbb5cc84a8f20c4d15b2a21" exitCode=0 Nov 01 22:24:06.377851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:06.377497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" event={"ID":"81975588-5e48-4d1b-b9b7-940bd0dfcee1","Type":"ContainerDied","Data":"1cfc38831a58f45c88c298ad5db3ad2c39a376bafdbb5cc84a8f20c4d15b2a21"} Nov 01 22:24:07.034959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.034910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:07.584466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.584423 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:07.708778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.708682 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\") pod \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " Nov 01 22:24:07.708950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.708852 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\" (UniqueName: \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\") pod \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " Nov 01 22:24:07.708950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.708934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\") pod \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\" (UID: \"81975588-5e48-4d1b-b9b7-940bd0dfcee1\") " Nov 01 22:24:07.709158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.709126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca" pod "81975588-5e48-4d1b-b9b7-940bd0dfcee1" (UID: "81975588-5e48-4d1b-b9b7-940bd0dfcee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:07.709614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.709589 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:07.711183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.711156 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd" pod "81975588-5e48-4d1b-b9b7-940bd0dfcee1" (UID: "81975588-5e48-4d1b-b9b7-940bd0dfcee1"). InnerVolumeSpecName "kube-api-access-9xppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:07.714799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.714769 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca" pod "81975588-5e48-4d1b-b9b7-940bd0dfcee1" (UID: "81975588-5e48-4d1b-b9b7-940bd0dfcee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:07.757109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.757072 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:07.810907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.810871 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xppd\" (UniqueName: \"kubernetes.io/projected/81975588-5e48-4d1b-b9b7-940bd0dfcee1-kube-api-access-9xppd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:07.810907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.810909 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81975588-5e48-4d1b-b9b7-940bd0dfcee1-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:07.826506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:07.826432 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:24:08.360308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.360282 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:24:08.402583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.402554 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" Nov 01 22:24:08.402583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.402568 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l" event={"ID":"81975588-5e48-4d1b-b9b7-940bd0dfcee1","Type":"ContainerDied","Data":"45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5"} Nov 01 22:24:08.402849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.402645 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45833031c4e261c8fa8f2b81c3c115552655b27113af21a6adc8cb7138486cb5" Nov 01 22:24:08.421214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.421109 5036 generic.go:356] "Generic (PLEG): container finished" podID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerID="b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0" exitCode=0 Nov 01 22:24:08.421214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.421153 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" Nov 01 22:24:08.421493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.421257 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerDied","Data":"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0"} Nov 01 22:24:08.421493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.421299 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2" event={"ID":"25475b42-d313-4ab9-9a89-f56cfb3028f6","Type":"ContainerDied","Data":"776a3879d1ae033d6450da59ec1ba5f21e4e28c65c8f3e210134ef782439dfce"} Nov 01 22:24:08.421493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.421326 5036 scope.go:122] "RemoveContainer" containerID="b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0" Nov 01 22:24:08.424219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.424128 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.424219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.424184 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.424402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.424223 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.424402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.424350 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.425761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.426237 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.426322 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.426360 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.426456 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.426773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.426548 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\") pod \"25475b42-d313-4ab9-9a89-f56cfb3028f6\" (UID: \"25475b42-d313-4ab9-9a89-f56cfb3028f6\") " Nov 01 22:24:08.427996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.427969 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.429487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.429461 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:08.429602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.429502 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.429706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.429687 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.429780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.429722 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.440931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.440867 5036 scope.go:122] "RemoveContainer" containerID="9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313" Nov 01 22:24:08.440931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.440912 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:24:08.453028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.453003 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:08.453157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.453045 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "kube-api-access-mr9rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:08.455219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.455201 5036 scope.go:122] "RemoveContainer" containerID="b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0" Nov 01 22:24:08.455507 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:08.455481 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0\": container with ID starting with b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0 not found: ID does not exist" containerID="b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0" Nov 01 22:24:08.455594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.455521 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0"} err="failed to get container status \"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0\": rpc error: code = NotFound desc = could not find container \"b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0\": container with ID starting with b410296a413f4f8596d372a136963c16d2a738ed6732db4f0b14f1816558d9c0 not found: ID does not exist" Nov 01 22:24:08.455594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.455544 5036 scope.go:122] "RemoveContainer" containerID="9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313" Nov 01 22:24:08.456002 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:08.455983 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313\": container with ID starting with 9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313 not found: ID does not exist" containerID="9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313" Nov 01 22:24:08.456107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.456005 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313"} err="failed to get container status \"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313\": rpc error: code = NotFound desc = could not find container \"9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313\": container with ID starting with 9a4130bcb2286c2bde2c0ae4e0bf51a905dd8af0da343697e951d56754d33313 not found: ID does not exist" Nov 01 22:24:08.456698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.456673 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy" pod "25475b42-d313-4ab9-9a89-f56cfb3028f6" (UID: "25475b42-d313-4ab9-9a89-f56cfb3028f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.460895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.460872 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:08.530119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530089 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530118 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr9rw\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-kube-api-access-mr9rw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530129 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530139 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25475b42-d313-4ab9-9a89-f56cfb3028f6-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530150 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530166 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530179 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.530187 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25475b42-d313-4ab9-9a89-f56cfb3028f6-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.611822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.611749 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:24:08.750353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.750311 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:24:08.754888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.754864 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-d2hv2"] Nov 01 22:24:08.954920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:08.954664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.098996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:09.098957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.348290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:09.348257 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:10.146146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:10.130418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:10.326890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:10.326851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:10.441827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:10.441725 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" path="/var/lib/kubelet/pods/25475b42-d313-4ab9-9a89-f56cfb3028f6/volumes" Nov 01 22:24:10.591487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:10.567317 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.121940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.121891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.242092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.242044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.464560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.464477 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:24:11.465555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465520 5036 cpu_manager.go:414] "containerMap: removing container" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.465555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465542 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.465749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465584 5036 cpu_manager.go:414] "containerMap: removing container" podUID="81975588-5e48-4d1b-b9b7-940bd0dfcee1" containerName="revoke-keys" Nov 01 22:24:11.465749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465592 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81975588-5e48-4d1b-b9b7-940bd0dfcee1" containerName="revoke-keys" Nov 01 22:24:11.465749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465605 5036 cpu_manager.go:414] "containerMap: removing container" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.465749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465614 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.465882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465775 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.465882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465793 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="81975588-5e48-4d1b-b9b7-940bd0dfcee1" containerName="revoke-keys" Nov 01 22:24:11.465882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.465805 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="25475b42-d313-4ab9-9a89-f56cfb3028f6" containerName="istio-proxy" Nov 01 22:24:11.472010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.471989 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.481427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.481397 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:24:11.482560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.481835 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-829cb0-openshift-default-dockercfg-d9lzp\"" Nov 01 22:24:11.539627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.539593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.539929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.539908 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540087 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ppj\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540214 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540364 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540640 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540689 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540742 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.540904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.540829 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.551483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.550524 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.660854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.647415 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.668707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.668663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.674466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.668933 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.674749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.674721 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.679671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.679641 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.688463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.679927 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.688768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.688745 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.688938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.688920 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ppj\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.693544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.693516 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.693822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.670719 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.712464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.670380 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.722558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.650436 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.722717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.686835 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.722816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.720290 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.722909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.722485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.779475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.776098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.779475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.776229 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.779475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.778188 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.791710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.785410 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ppj\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\") pod \"e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:11.837763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:11.837724 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:12.029702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.029675 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh"] Nov 01 22:24:12.045506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.036103 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8a249fd-v8mxh"] Nov 01 22:24:12.229973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.229936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:12.300850 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:24:12.300807 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59073435_6a6e_462a_a5d1_44aec4513092.slice/crio-4138d58a8228d892d8b8615a14db9ca27e7bac25875d5202fd05d82c1bcc0ca2 WatchSource:0}: Error finding container 4138d58a8228d892d8b8615a14db9ca27e7bac25875d5202fd05d82c1bcc0ca2: Status 404 returned error can't find the container with id 4138d58a8228d892d8b8615a14db9ca27e7bac25875d5202fd05d82c1bcc0ca2 Nov 01 22:24:12.305651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.303039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:24:12.325480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.323570 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:12.325480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.323663 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:12.325480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.323708 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:12.440827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.440799 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172054ca-402d-4bf7-af9e-132dbacede8e" path="/var/lib/kubelet/pods/172054ca-402d-4bf7-af9e-132dbacede8e/volumes" Nov 01 22:24:12.468592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.468552 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerStarted","Data":"c29ef76b87ed32778b7557552cbad1084021a6cd447e04d8e74e0b095f7b5a5c"} Nov 01 22:24:12.468786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.468633 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerStarted","Data":"4138d58a8228d892d8b8615a14db9ca27e7bac25875d5202fd05d82c1bcc0ca2"} Nov 01 22:24:12.505465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.502525 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podStartSLOduration=1.502506672 podStartE2EDuration="1.502506672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:12.490957209 +0000 UTC m=+2406.183937698" watchObservedRunningTime="2026-11-01 22:24:12.502506672 +0000 UTC m=+2406.195487159" Nov 01 22:24:12.838784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:12.838756 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:13.520400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:13.520361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:13.699760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:13.699720 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:13.845394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:13.845315 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:14.240562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.237258 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:24:14.308961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.308921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:14.570936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.567535 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:14.570936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.570404 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:24:14.751517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.751429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:14.890028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:14.889941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:15.924333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:15.924290 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:16.052279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:16.052190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:16.224203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:16.213036 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:17.101535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:17.101484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:17.235914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:17.235874 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:17.355532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:17.353594 5036 scope.go:122] "RemoveContainer" containerID="9036344dda94c6dfbdc085117f52bfd6fc5a96e2ffbda9921637743819ecebdc" Nov 01 22:24:17.371312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:17.371277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:18.117407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:18.117368 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:18.860686 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:18.860641 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:24:18.898698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:18.898662 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:19.649765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:19.649727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:20.463584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.463542 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:20.682463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.680792 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:24:20.694410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.694380 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:24:20.695349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.694509 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.706737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.706707 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:24:20.707462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.707375 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.713327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.713300 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:24:20.768644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.768612 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e25317-openshift-default-dockercfg-v8cbj\"" Nov 01 22:24:20.823301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823257 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823366 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823434 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823537 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7dm\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.823675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823568 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823719 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.823798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823747 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.823798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823781 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823868 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.823999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.823979 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.824038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824007 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.824038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824024 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.824107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhv7\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.824143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824122 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.824203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824188 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.824247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.824247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.824237 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.925990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926051 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926176 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926254 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhv7\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926285 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926385 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926485 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926534 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926566 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926614 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926707 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926786 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926826 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7dm\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.926919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926852 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.926919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.926905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927054 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.927143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927101 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.927205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927164 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927206 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927232 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927054 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927285 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927369 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.927857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927782 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.927963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.927894 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.928297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.928269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.928421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.928327 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.929363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.929336 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.929660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.929640 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.930098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.930072 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.930210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.930166 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.938048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.938020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.938518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.938496 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:20.938618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.938599 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhv7\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:20.939123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:20.939105 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7dm\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:21.021662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.021633 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:21.037282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.037255 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:21.231004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.230954 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:21.338912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.337968 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:24:21.338912 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:24:21.338895 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0e0635_0e28_4f51_8f97_349b69c0f3f8.slice/crio-7093f0969d98c54fcf5ae74426f383b3553fa307d1f5267526893d63329dbefe WatchSource:0}: Error finding container 7093f0969d98c54fcf5ae74426f383b3553fa307d1f5267526893d63329dbefe: Status 404 returned error can't find the container with id 7093f0969d98c54fcf5ae74426f383b3553fa307d1f5267526893d63329dbefe Nov 01 22:24:21.342101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.342067 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.342243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.342135 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.342243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.342165 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.357136 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:24:21.357113 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc28e919_b89c_4a2e_b221_2cf7ffc73448.slice/crio-da687c11d72da60b660db4114546b6d770dd9a4a3dc2f57364f65b455e393ca7 WatchSource:0}: Error finding container da687c11d72da60b660db4114546b6d770dd9a4a3dc2f57364f65b455e393ca7: Status 404 returned error can't find the container with id da687c11d72da60b660db4114546b6d770dd9a4a3dc2f57364f65b455e393ca7 Nov 01 22:24:21.359788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.359757 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.359873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.359817 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.359873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.359847 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:24:21.361040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.361015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:24:21.641580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.641475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" event={"ID":"cc28e919-b89c-4a2e-b221-2cf7ffc73448","Type":"ContainerStarted","Data":"56056fac195946d8b1e19875a984f1226432cf0f1599d83dd84f2dcacf21a93a"} Nov 01 22:24:21.641580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.641553 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" event={"ID":"cc28e919-b89c-4a2e-b221-2cf7ffc73448","Type":"ContainerStarted","Data":"da687c11d72da60b660db4114546b6d770dd9a4a3dc2f57364f65b455e393ca7"} Nov 01 22:24:21.643810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.643770 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" event={"ID":"6f0e0635-0e28-4f51-8f97-349b69c0f3f8","Type":"ContainerStarted","Data":"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018"} Nov 01 22:24:21.643980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.643834 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" event={"ID":"6f0e0635-0e28-4f51-8f97-349b69c0f3f8","Type":"ContainerStarted","Data":"7093f0969d98c54fcf5ae74426f383b3553fa307d1f5267526893d63329dbefe"} Nov 01 22:24:21.672912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.672853 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podStartSLOduration=1.672835492 podStartE2EDuration="1.672835492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:21.663687805 +0000 UTC m=+2415.356668293" watchObservedRunningTime="2026-11-01 22:24:21.672835492 +0000 UTC m=+2415.365815979" Nov 01 22:24:21.691668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:21.691619 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podStartSLOduration=1.6916017509999999 podStartE2EDuration="1.691601751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:21.686517916 +0000 UTC m=+2415.379498405" watchObservedRunningTime="2026-11-01 22:24:21.691601751 +0000 UTC m=+2415.384582238" Nov 01 22:24:22.024468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.023916 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:22.024468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.024133 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:22.024468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.024182 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:22.043198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.039543 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:22.043198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.039726 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:22.043198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.039778 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:22.085168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.074919 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:22.956467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:22.951619 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:23.039023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:23.031084 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:23.039023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:23.031191 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:23.064536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:23.063373 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:23.064536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:23.063432 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:23.843280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:23.843244 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:24.024469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:24.023851 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:24.024469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:24.023927 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:24.038483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:24.037996 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:24.038483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:24.038049 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:24.556468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:24.553477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:25.039587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.025588 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:25.039587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.025660 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:25.039587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.039069 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:25.039587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.039137 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:25.286437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.286330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:25.832470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.729763 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:24:25.856466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:25.848997 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:26.026295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.025861 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:26.026295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.025923 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:26.037912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.037795 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:26.037912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.037849 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:26.327505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.326005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:26.571200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.571161 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:26.757415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:26.757318 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:27.029471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.024952 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:27.029471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.025053 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:27.093019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.089149 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:27.093019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.089211 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:27.323015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.322917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:27.468116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:27.468063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:28.023868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.023828 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 01 22:24:28.024132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.024100 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 01 22:24:28.038827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.038797 5036 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 01 22:24:28.039099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.039053 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 01 22:24:28.054063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.054031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:28.825573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:28.825521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:29.031016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.030972 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:29.032821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.032799 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:29.034309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.034292 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:24:29.048164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.048139 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:29.048665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.048639 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:29.050423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.050400 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:24:29.277595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.277547 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" containerID="cri-o://cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd" gracePeriod=45 Nov 01 22:24:29.621487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.621428 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:24:29.737867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.737829 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:29.744904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.744872 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.745056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.744974 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.745056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745008 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.745178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745147 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\" (UniqueName: \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.745538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745187 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:29.745626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745203 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.745725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745704 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\") pod \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\" (UID: \"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f\") " Nov 01 22:24:29.746578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.745306 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:29.746787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.746769 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.746850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.746793 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.746850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.746812 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:29.749022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.748996 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:29.749157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.749066 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "kube-api-access-gq2b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:29.749157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.749086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm" pod "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" (UID: "081bc1e8-b67d-4ba0-80f6-cb5367e44a6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:29.848418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.848333 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.848418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.848366 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.848418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.848388 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.848418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.848398 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2b7\" (UniqueName: \"kubernetes.io/projected/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f-kube-api-access-gq2b7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.862884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:29.862837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.003787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.003751 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.196914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.196818 5036 generic.go:356] "Generic (PLEG): container finished" podID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerID="cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd" exitCode=0 Nov 01 22:24:30.196914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.196867 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" Nov 01 22:24:30.197141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.196928 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" event={"ID":"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f","Type":"ContainerDied","Data":"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd"} Nov 01 22:24:30.197141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.196980 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr" event={"ID":"081bc1e8-b67d-4ba0-80f6-cb5367e44a6f","Type":"ContainerDied","Data":"73cf32b7eff7c135a313b616927722a45e491bd6165878c313022d84d0c9b661"} Nov 01 22:24:30.197141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.197007 5036 scope.go:122] "RemoveContainer" containerID="cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd" Nov 01 22:24:30.212629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.212605 5036 scope.go:122] "RemoveContainer" containerID="cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd" Nov 01 22:24:30.213932 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:30.213873 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd\": container with ID starting with cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd not found: ID does not exist" containerID="cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd" Nov 01 22:24:30.213932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.213912 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd"} err="failed to get container status \"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd\": rpc error: code = NotFound desc = could not find container \"cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd\": container with ID starting with cf09ef6159a99024fe9299edbe8752049a7708fa08c862e1dc849913f13665cd not found: ID does not exist" Nov 01 22:24:30.230633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.230608 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:24:30.238493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.238466 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e3890e/sub-test-model-e3890e-kserve-fb6f5fc97-x6xzr"] Nov 01 22:24:30.375995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.375952 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.444492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.444395 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" path="/var/lib/kubelet/pods/081bc1e8-b67d-4ba0-80f6-cb5367e44a6f/volumes" Nov 01 22:24:30.547466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.541092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.956524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:30.952103 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:31.574774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:31.574740 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:31.772564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:31.772526 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:32.077187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:32.077141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:32.925002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:32.924965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:33.708515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:33.708467 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:34.614071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:34.614034 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:35.609152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:35.609106 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.255405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.255367 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:24:36.256567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.256540 5036 cpu_manager.go:414] "containerMap: removing container" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" Nov 01 22:24:36.256750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.256728 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" Nov 01 22:24:36.257188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.257162 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="081bc1e8-b67d-4ba0-80f6-cb5367e44a6f" containerName="main" Nov 01 22:24:36.262402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.262371 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.267129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.267106 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-829cb0-serving-cert\"" Nov 01 22:24:36.290492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.286697 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:24:36.344461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.343602 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.356696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.356493 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.356696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.356551 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2lr9\" (UniqueName: \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.356930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.356697 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.362601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.362338 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.464357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.464312 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.464573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.464395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2lr9\" (UniqueName: \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.464573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.464538 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.464713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.464590 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.465113 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:36.464953 5036 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-serving-cert: secret "maas-api-e2e-ipp-b-829cb0-serving-cert" not found Nov 01 22:24:36.465766 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:36.465737 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls podName:9173fec5-0a69-4315-93dd-3791c90ce8be nodeName:}" failed. No retries permitted until 2026-11-01 22:24:36.96500856 +0000 UTC m=+2430.657989038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls") pod "maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" (UID: "9173fec5-0a69-4315-93dd-3791c90ce8be") : secret "maas-api-e2e-ipp-b-829cb0-serving-cert" not found Nov 01 22:24:36.465982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.465926 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.469017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.468992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.471287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.471259 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.478268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.478242 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2lr9\" (UniqueName: \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:36.601238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.601203 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.979248 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:36.976023 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:37.018467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.012596 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") pod \"maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:37.151270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.151225 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:37.200476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.196890 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:37.550128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.550092 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:24:37.552340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.552314 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:24:37.943806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:37.943723 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:38.285073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:38.285031 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" event={"ID":"9173fec5-0a69-4315-93dd-3791c90ce8be","Type":"ContainerStarted","Data":"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723"} Nov 01 22:24:38.285525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:38.285105 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" event={"ID":"9173fec5-0a69-4315-93dd-3791c90ce8be","Type":"ContainerStarted","Data":"f819b7396a3f26d4209c47ee83bd2b22135df559156afdd37b52a0dd0981d2e1"} Nov 01 22:24:38.285525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:38.285137 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:38.307753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:38.307689 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" podStartSLOduration=1.8840848270000001 podStartE2EDuration="2.307670656s" totalImagesPullingTime="423.585829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:38.305967803 +0000 UTC m=+2431.998948291" watchObservedRunningTime="2026-11-01 22:24:38.307670656 +0000 UTC m=+2432.000651143" Nov 01 22:24:39.028356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:39.028323 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:39.944756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:39.944712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:40.715305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:40.715265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:41.104699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.104653 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" containerID="cri-o://d9a5be8eb9b458fce8a79b60874a771aac5cacd2a53643d00fd0c543b65a0b2c" gracePeriod=45 Nov 01 22:24:41.339367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.338959 5036 generic.go:356] "Generic (PLEG): container finished" podID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerID="d9a5be8eb9b458fce8a79b60874a771aac5cacd2a53643d00fd0c543b65a0b2c" exitCode=0 Nov 01 22:24:41.339367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.339291 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" event={"ID":"e62cd799-9ae1-4b43-8552-0902237db3a4","Type":"ContainerDied","Data":"d9a5be8eb9b458fce8a79b60874a771aac5cacd2a53643d00fd0c543b65a0b2c"} Nov 01 22:24:41.351151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.351117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:41.500342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.500300 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:24:41.594929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.594892 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595161 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.595263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595204 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595272 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\" (UniqueName: \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595323 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595363 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.595556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595407 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\") pod \"e62cd799-9ae1-4b43-8552-0902237db3a4\" (UID: \"e62cd799-9ae1-4b43-8552-0902237db3a4\") " Nov 01 22:24:41.595617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.595589 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.596370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.596343 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.596548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.596376 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.596548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.596393 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.598067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.598040 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "kube-api-access-csmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:41.598172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.598161 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.598845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.598818 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm" pod "e62cd799-9ae1-4b43-8552-0902237db3a4" (UID: "e62cd799-9ae1-4b43-8552-0902237db3a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.698133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.698103 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62cd799-9ae1-4b43-8552-0902237db3a4-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.698133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.698130 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csmws\" (UniqueName: \"kubernetes.io/projected/e62cd799-9ae1-4b43-8552-0902237db3a4-kube-api-access-csmws\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.698319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:41.698140 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62cd799-9ae1-4b43-8552-0902237db3a4-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:42.046641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.046608 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:42.351723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.351644 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" event={"ID":"e62cd799-9ae1-4b43-8552-0902237db3a4","Type":"ContainerDied","Data":"246a72653781f0a55c9e7108746e0f0108555d039e17d0aab6b6cf96779a8196"} Nov 01 22:24:42.351723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.351667 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh" Nov 01 22:24:42.351723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.351692 5036 scope.go:122] "RemoveContainer" containerID="d9a5be8eb9b458fce8a79b60874a771aac5cacd2a53643d00fd0c543b65a0b2c" Nov 01 22:24:42.378187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.378158 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:24:42.383013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.382985 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-51e3b7/sub-test-model-51e3b7-kserve-55c75999c5-2sghh"] Nov 01 22:24:42.447237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.447202 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" path="/var/lib/kubelet/pods/e62cd799-9ae1-4b43-8552-0902237db3a4/volumes" Nov 01 22:24:42.517397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.517185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:42.644052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.643968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:42.776473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:42.775982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:43.016350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:43.016245 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:43.706314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:43.706278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:44.316164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:44.316137 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:24:44.367637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:44.367598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:45.078471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:45.064593 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:45.713837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:45.713799 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:46.365838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:46.365806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:47.189843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:47.189796 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:47.848938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:47.848900 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:48.531990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:48.531948 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:48.859951 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:24:48.859867 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:24:49.245556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.245439 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:49.945113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.945080 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:24:49.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.945905 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" Nov 01 22:24:49.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.945927 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" Nov 01 22:24:49.946126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.946090 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cd799-9ae1-4b43-8552-0902237db3a4" containerName="main" Nov 01 22:24:49.951958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.951936 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:49.962986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.959938 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0bb375-serving-cert\"" Nov 01 22:24:49.962986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:49.962944 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:24:50.015385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.015345 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdpl\" (UniqueName: \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.015715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.015693 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.015770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.015750 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.015848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.015835 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.020051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.020023 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:50.117162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.117131 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.117375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.117177 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.117375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.117257 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.117375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.117333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdpl\" (UniqueName: \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.118107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.118074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.120119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.120092 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.120273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.120207 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.130126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.130095 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdpl\" (UniqueName: \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\") pod \"maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.166471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.160068 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:50.277760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.277728 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:50.723146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.722990 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:24:50.854856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:50.854814 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:51.620475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:51.619306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" event={"ID":"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd","Type":"ContainerStarted","Data":"2e760f8d8019d86eb4eb66bd8c36247e41b16a2718f3ee0ec0a0d1f2a0c398d2"} Nov 01 22:24:51.620475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:51.619373 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" event={"ID":"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd","Type":"ContainerStarted","Data":"932617b1fea871c6cff421bb845ea9006583ff68b5e4332478f067da06ebd6d0"} Nov 01 22:24:51.621353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:51.621282 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:51.902253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:51.902156 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:52.714664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:52.714627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:53.551707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:53.551669 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:54.286497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:54.283595 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:54.887545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:54.872620 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:55.039033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:55.033754 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:55.574548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:55.573500 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:56.827469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:56.815631 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:57.736815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:57.736779 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:58.531467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:58.527893 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:58.666906 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:58.666873 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:24:58.714470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:58.709104 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" podStartSLOduration=9.262034915 podStartE2EDuration="9.709085093s" totalImagesPullingTime="447.050178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:51.685826885 +0000 UTC m=+2445.378807387" watchObservedRunningTime="2026-11-01 22:24:58.709085093 +0000 UTC m=+2452.402065580" Nov 01 22:24:59.395354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.395312 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x"] Nov 01 22:24:59.401612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.401585 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.412107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.407240 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:24:59.412107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.407367 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:24:59.413809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.413768 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x"] Nov 01 22:24:59.418096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.418070 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:59.513704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.513665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.514135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.514108 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.514315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.514297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.615721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.615685 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.616142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.615807 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.616142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.615840 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.616696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.616671 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.618609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.618580 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.628039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.628009 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\") pod \"maas-api-revoke-sub-fdb31daa-kqt2x\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.828616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.828575 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:24:59.930269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:24:59.930089 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:00.080253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.080163 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:00.134854 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:00.134826 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaa966a_64d9_403a_9485_b5f2d280b33a.slice/crio-012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5 WatchSource:0}: Error finding container 012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5: Status 404 returned error can't find the container with id 012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5 Nov 01 22:25:00.135083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.135035 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x"] Nov 01 22:25:00.398856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.398689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" event={"ID":"ecaa966a-64d9-403a-9485-b5f2d280b33a","Type":"ContainerStarted","Data":"012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5"} Nov 01 22:25:00.517600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.517241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:00.646195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.646111 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:00.772050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:00.772017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:01.410593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:01.410563 5036 generic.go:356] "Generic (PLEG): container finished" podID="ecaa966a-64d9-403a-9485-b5f2d280b33a" containerID="b84f9394aaaafbcd3c35e6acc31015a01fea214be6c5d180bd5b15f879f21bcd" exitCode=0 Nov 01 22:25:01.410729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:01.410624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" event={"ID":"ecaa966a-64d9-403a-9485-b5f2d280b33a","Type":"ContainerDied","Data":"b84f9394aaaafbcd3c35e6acc31015a01fea214be6c5d180bd5b15f879f21bcd"} Nov 01 22:25:01.556626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:01.556591 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:02.338273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.338234 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:02.625426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.625400 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:25:02.773893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.773863 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\") pod \"ecaa966a-64d9-403a-9485-b5f2d280b33a\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " Nov 01 22:25:02.774075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.773960 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\") pod \"ecaa966a-64d9-403a-9485-b5f2d280b33a\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " Nov 01 22:25:02.774075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.774000 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\") pod \"ecaa966a-64d9-403a-9485-b5f2d280b33a\" (UID: \"ecaa966a-64d9-403a-9485-b5f2d280b33a\") " Nov 01 22:25:02.774385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.774358 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca" pod "ecaa966a-64d9-403a-9485-b5f2d280b33a" (UID: "ecaa966a-64d9-403a-9485-b5f2d280b33a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:02.774738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.774713 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:02.776144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.776119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca" pod "ecaa966a-64d9-403a-9485-b5f2d280b33a" (UID: "ecaa966a-64d9-403a-9485-b5f2d280b33a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:02.776294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.776276 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b" pod "ecaa966a-64d9-403a-9485-b5f2d280b33a" (UID: "ecaa966a-64d9-403a-9485-b5f2d280b33a"). InnerVolumeSpecName "kube-api-access-jl52b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:02.876947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.876871 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa966a-64d9-403a-9485-b5f2d280b33a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:02.876947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:02.876910 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/ecaa966a-64d9-403a-9485-b5f2d280b33a-kube-api-access-jl52b\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:03.013343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:03.013304 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:03.432943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:03.432904 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" event={"ID":"ecaa966a-64d9-403a-9485-b5f2d280b33a","Type":"ContainerDied","Data":"012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5"} Nov 01 22:25:03.433341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:03.432947 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x" Nov 01 22:25:03.433341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:03.432959 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012290baf271a00f7702d1867e3e485d3b68c88bcc831daa4af2bc0760a99bd5" Nov 01 22:25:03.746664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:03.746563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:04.429359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:04.429326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:05.070371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:05.070330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:05.838847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:05.838809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:06.740078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:06.740045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:07.480518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:07.475266 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:08.219473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:08.219415 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:09.074316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.074276 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:09.599395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.599357 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4"] Nov 01 22:25:09.600503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.600479 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ecaa966a-64d9-403a-9485-b5f2d280b33a" containerName="revoke-keys" Nov 01 22:25:09.600590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.600505 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaa966a-64d9-403a-9485-b5f2d280b33a" containerName="revoke-keys" Nov 01 22:25:09.600754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.600738 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaa966a-64d9-403a-9485-b5f2d280b33a" containerName="revoke-keys" Nov 01 22:25:09.605309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.605284 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.612957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.612931 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:25:09.613934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.613887 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:25:09.623793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.623767 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4"] Nov 01 22:25:09.699700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.699661 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.700018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.699999 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.700074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.700060 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.801567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.801529 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.801898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.801872 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.801970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.801959 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.802731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.802699 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.804256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.804231 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.817016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.816987 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") pod \"maas-api-revoke-sub-f2096d7c-4bzd4\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:09.924297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.924204 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:09.950228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:09.949768 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:10.270481 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:10.270412 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9571de_d00a_4849_a3f9_e2b7feebd067.slice/crio-70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6 WatchSource:0}: Error finding container 70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6: Status 404 returned error can't find the container with id 70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6 Nov 01 22:25:10.271266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:10.271240 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4"] Nov 01 22:25:10.545082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:10.545045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerStarted","Data":"70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6"} Nov 01 22:25:10.644001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:10.643968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:11.438003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:11.437969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:11.558120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:11.558086 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerStarted","Data":"d952b492084cd7514202888c41bf00b54ae452fd285491a10ee2a43351e961dd"} Nov 01 22:25:11.580617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:11.580567 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" podStartSLOduration=2.580553256 podStartE2EDuration="2.580553256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:11.57732809 +0000 UTC m=+2465.270308577" watchObservedRunningTime="2026-11-01 22:25:11.580553256 +0000 UTC m=+2465.273533773" Nov 01 22:25:12.182149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:12.182110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:13.193297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:13.193253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:13.749927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:13.749888 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:13.878252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:13.878200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:14.475156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:14.475104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:15.245656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:15.245624 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:15.948181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:15.948143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:16.087089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:16.087055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:16.287465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:16.281316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:17.434439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:17.434343 5036 scope.go:122] "RemoveContainer" containerID="2d382f15d4ae9bf7d089643b45a3c0488b204d3e435450466137f1c2e9a62083" Nov 01 22:25:17.445810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:17.445791 5036 scope.go:122] "RemoveContainer" containerID="a143ed25abc126efd440c79be62adef3d26849b840e5b20e075c758430e5e739" Nov 01 22:25:18.860388 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:25:18.860350 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:25:22.332870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:22.332826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:23.257823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:23.257776 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:24.059075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:24.059039 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:25.392060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:25.392020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:25.884700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:25.884662 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:26.015726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:26.015686 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:26.744871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:26.744820 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:27.551393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:27.551353 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:28.568972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:28.568939 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:29.377212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:29.377174 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:30.125783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:30.125743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:31.012088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:31.012042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:31.740064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:31.740031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:32.425303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:32.425264 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:33.142251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:33.142211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:33.971376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:33.971330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:34.661465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.656796 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:34.730799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.730765 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb"] Nov 01 22:25:34.735615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.735595 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.745614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.745581 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb"] Nov 01 22:25:34.772496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.772425 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.772630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.772561 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.772710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.772628 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.873929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.873895 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.874098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.873957 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.874098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.874000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.874652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.874630 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.876344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.876319 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:34.887160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:34.887128 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") pod \"maas-api-revoke-sub-fb860254-4hmqb\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:35.059939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:35.059907 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:35.349642 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:35.349604 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84f35aa_3b65_4a84_a0a8_0d33ff418a1f.slice/crio-ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7 WatchSource:0}: Error finding container ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7: Status 404 returned error can't find the container with id ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7 Nov 01 22:25:35.351614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:35.351584 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb"] Nov 01 22:25:35.427989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:35.427958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:35.800192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:35.800155 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerStarted","Data":"ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7"} Nov 01 22:25:36.150231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:36.150152 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:36.813641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:36.813600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerStarted","Data":"bc55dfc2ea3116908aa81bd555b59a90e97c0ac7211ad330503b9b354d0dfa14"} Nov 01 22:25:36.817380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:36.817349 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:36.860466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:36.860147 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" podStartSLOduration=2.8601136609999998 podStartE2EDuration="2.860113661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:36.856095281 +0000 UTC m=+2490.549075765" watchObservedRunningTime="2026-11-01 22:25:36.860113661 +0000 UTC m=+2490.553094148" Nov 01 22:25:38.528576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:38.528540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:39.311151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:39.311112 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:39.955581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:39.955540 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:41.866888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:41.866849 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerID="d952b492084cd7514202888c41bf00b54ae452fd285491a10ee2a43351e961dd" exitCode=28 Nov 01 22:25:41.867405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:41.866925 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerDied","Data":"d952b492084cd7514202888c41bf00b54ae452fd285491a10ee2a43351e961dd"} Nov 01 22:25:41.867520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:41.867496 5036 scope.go:122] "RemoveContainer" containerID="d952b492084cd7514202888c41bf00b54ae452fd285491a10ee2a43351e961dd" Nov 01 22:25:42.877949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:42.877913 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerStarted","Data":"9b7a352cc7505613ea32ea7fc6fc1ad3c2876a654b47e893c70c458c6a6295f5"} Nov 01 22:25:44.375295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:44.375247 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:25:46.376924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:46.376886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:46.951768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:46.951719 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:47.112912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:47.112868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:47.741322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:47.741290 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:48.866509 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:25:48.866468 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:25:49.134947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:49.134857 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:49.859517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:49.859480 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:50.549881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:50.549849 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:51.231430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:51.231390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:52.021381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:52.021221 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:52.857645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:52.857590 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:53.021159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:53.021122 5036 generic.go:356] "Generic (PLEG): container finished" podID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerID="bc55dfc2ea3116908aa81bd555b59a90e97c0ac7211ad330503b9b354d0dfa14" exitCode=35 Nov 01 22:25:53.021361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:53.021195 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerDied","Data":"bc55dfc2ea3116908aa81bd555b59a90e97c0ac7211ad330503b9b354d0dfa14"} Nov 01 22:25:53.021713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:53.021693 5036 scope.go:122] "RemoveContainer" containerID="bc55dfc2ea3116908aa81bd555b59a90e97c0ac7211ad330503b9b354d0dfa14" Nov 01 22:25:53.547755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:53.547558 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:54.036585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:54.036541 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerStarted","Data":"5e2c2e21748feea21c03cd706246537ee17a54ff73af5ea215ef4f5faa81fc2f"} Nov 01 22:25:54.535819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:54.535780 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:55.264834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:55.264783 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:56.124021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:56.123977 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:56.756384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:56.756348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:57.065744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.065712 5036 generic.go:356] "Generic (PLEG): container finished" podID="59073435-6a6e-462a-a5d1-44aec4513092" containerID="c29ef76b87ed32778b7557552cbad1084021a6cd447e04d8e74e0b095f7b5a5c" exitCode=0 Nov 01 22:25:57.065912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.065755 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerDied","Data":"c29ef76b87ed32778b7557552cbad1084021a6cd447e04d8e74e0b095f7b5a5c"} Nov 01 22:25:57.066212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.066195 5036 scope.go:122] "RemoveContainer" containerID="c29ef76b87ed32778b7557552cbad1084021a6cd447e04d8e74e0b095f7b5a5c" Nov 01 22:25:57.068119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.068089 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:25:57.068228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.068164 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:25:57.068228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.068193 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:25:57.529607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:57.529577 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:58.096467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.087520 5036 generic.go:356] "Generic (PLEG): container finished" podID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerID="5e2c2e21748feea21c03cd706246537ee17a54ff73af5ea215ef4f5faa81fc2f" exitCode=0 Nov 01 22:25:58.096467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.087732 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerDied","Data":"5e2c2e21748feea21c03cd706246537ee17a54ff73af5ea215ef4f5faa81fc2f"} Nov 01 22:25:58.096467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.087816 5036 scope.go:122] "RemoveContainer" containerID="bc55dfc2ea3116908aa81bd555b59a90e97c0ac7211ad330503b9b354d0dfa14" Nov 01 22:25:58.114513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.112497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerStarted","Data":"df631f6682a0c274fc38cf510beb860a574cdab8164570b6c253b72872f78fd8"} Nov 01 22:25:58.115993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.115963 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerID="9b7a352cc7505613ea32ea7fc6fc1ad3c2876a654b47e893c70c458c6a6295f5" exitCode=0 Nov 01 22:25:58.116153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.116045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerDied","Data":"9b7a352cc7505613ea32ea7fc6fc1ad3c2876a654b47e893c70c458c6a6295f5"} Nov 01 22:25:58.121714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.121687 5036 scope.go:122] "RemoveContainer" containerID="d952b492084cd7514202888c41bf00b54ae452fd285491a10ee2a43351e961dd" Nov 01 22:25:58.242185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.242086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:58.838598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.838549 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:25:58.839728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.839699 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Nov 01 22:25:58.839895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.839767 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Nov 01 22:25:58.900469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:58.891008 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:59.432559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.432531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:25:59.467621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.467583 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:25:59.548737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.548698 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\") pod \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " Nov 01 22:25:59.548935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.548880 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\") pod \"8e9571de-d00a-4849-a3f9-e2b7feebd067\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " Nov 01 22:25:59.548935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.548923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") pod \"8e9571de-d00a-4849-a3f9-e2b7feebd067\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " Nov 01 22:25:59.549058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.549031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") pod \"8e9571de-d00a-4849-a3f9-e2b7feebd067\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " Nov 01 22:25:59.549268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.549237 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") pod \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " Nov 01 22:25:59.551348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.549506 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") pod \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " Nov 01 22:25:59.551348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.550000 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca" pod "8e9571de-d00a-4849-a3f9-e2b7feebd067" (UID: "8e9571de-d00a-4849-a3f9-e2b7feebd067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:59.551348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.550362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca" pod "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" (UID: "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:59.551348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.551183 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.551348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.551218 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.588467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.568228 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:25:59.660633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.649300 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:59.667122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.666189 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca" pod "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" (UID: "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:59.667574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.667524 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7" pod "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" (UID: "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f"). InnerVolumeSpecName "kube-api-access-9h5l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:59.667896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.667851 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5" pod "8e9571de-d00a-4849-a3f9-e2b7feebd067" (UID: "8e9571de-d00a-4849-a3f9-e2b7feebd067"). InnerVolumeSpecName "kube-api-access-z2rd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:59.671845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.671597 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca" pod "8e9571de-d00a-4849-a3f9-e2b7feebd067" (UID: "8e9571de-d00a-4849-a3f9-e2b7feebd067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:59.674531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.674415 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") pod \"8e9571de-d00a-4849-a3f9-e2b7feebd067\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " Nov 01 22:25:59.675340 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:59.674914 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8e9571de-d00a-4849-a3f9-e2b7feebd067/volumes/kubernetes.io~projected/kube-api-access-z2rd5 Nov 01 22:25:59.675340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.674920 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") pod \"8e9571de-d00a-4849-a3f9-e2b7feebd067\" (UID: \"8e9571de-d00a-4849-a3f9-e2b7feebd067\") " Nov 01 22:25:59.675340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.674948 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5" pod "8e9571de-d00a-4849-a3f9-e2b7feebd067" (UID: "8e9571de-d00a-4849-a3f9-e2b7feebd067"). InnerVolumeSpecName "kube-api-access-z2rd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:59.675340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.675079 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") pod \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " Nov 01 22:25:59.675340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.675113 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") pod \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\" (UID: \"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f\") " Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:59.675415 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f/volumes/kubernetes.io~projected/kube-api-access-9h5l7 Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.675479 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7" pod "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" (UID: "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f"). InnerVolumeSpecName "kube-api-access-9h5l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:59.675536 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8e9571de-d00a-4849-a3f9-e2b7feebd067/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.675545 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca" pod "8e9571de-d00a-4849-a3f9-e2b7feebd067" (UID: "8e9571de-d00a-4849-a3f9-e2b7feebd067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:25:59.675588 5036 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 01 22:25:59.675678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.675596 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca" pod "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" (UID: "f84f35aa-3b65-4a84-a0a8-0d33ff418a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:59.676430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.676403 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.676553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.676432 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h5l7\" (UniqueName: \"kubernetes.io/projected/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f-kube-api-access-9h5l7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.676553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.676473 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rd5\" (UniqueName: \"kubernetes.io/projected/8e9571de-d00a-4849-a3f9-e2b7feebd067-kube-api-access-z2rd5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.676553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.676488 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9571de-d00a-4849-a3f9-e2b7feebd067-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:59.839177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.839143 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Nov 01 22:25:59.839365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:25:59.839195 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Nov 01 22:26:00.145677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.145594 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" Nov 01 22:26:00.145848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.145595 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb" event={"ID":"f84f35aa-3b65-4a84-a0a8-0d33ff418a1f","Type":"ContainerDied","Data":"ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7"} Nov 01 22:26:00.145848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.145746 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab33380029cab9272faeaafdc801577aed9990a82cf5ef9971897bd2d68e57d7" Nov 01 22:26:00.147908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.147883 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" Nov 01 22:26:00.148597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.148519 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4" event={"ID":"8e9571de-d00a-4849-a3f9-e2b7feebd067","Type":"ContainerDied","Data":"70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6"} Nov 01 22:26:00.148597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.148571 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c8d3405dd95bbe1161a583277d710494d72dc9bf5e5ee6982e663d0c5801c6" Nov 01 22:26:00.320467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.317140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:00.751326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.751285 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:00.839332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.839296 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Nov 01 22:26:00.839550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.839347 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Nov 01 22:26:00.958896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:00.958862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:01.638493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:01.633546 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:01.838830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:01.838797 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Nov 01 22:26:01.839234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:01.838847 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Nov 01 22:26:02.449164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:02.449130 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:02.839655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:02.839625 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Nov 01 22:26:02.840046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:02.839675 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Nov 01 22:26:03.132558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.132475 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:03.409400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.409308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:03.778764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.778731 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:26:03.780065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780029 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:03.780065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780061 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:03.780295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780079 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:03.780295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780086 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:03.780417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780317 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:03.780417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780343 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:03.780417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.780356 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:03.787242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.784975 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.799839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.799701 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a00c6a-openshift-default-dockercfg-hwmjz\"" Nov 01 22:26:03.830249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.829556 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:26:03.833032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833002 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833050 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833175 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833227 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28j5\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833381 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833461 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.833548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.833498 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.857568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.856767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:03.866794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.866771 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:26:03.869258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.869238 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.962369 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.962436 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.962920 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.962977 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963098 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963219 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28j5\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963391 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.963482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.964272 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.964321 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.964999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.964587 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.965921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.965726 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.966598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.966538 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:03.970474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:03.969976 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:04.002641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.002611 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:04.015640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.015613 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:04.017897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.017873 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28j5\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\") pod \"e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:04.140754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.140650 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:04.150969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.150942 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:04.641130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.641072 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:26:04.647653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.647611 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:04.647787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.647705 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:04.647787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.647746 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:04.662911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.662853 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:04.875202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.875164 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s"] Nov 01 22:26:04.876206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.876180 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:04.876343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.876221 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:04.876343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.876247 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:04.876343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.876256 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" containerName="revoke-keys" Nov 01 22:26:04.876554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.876537 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" containerName="revoke-keys" Nov 01 22:26:04.880533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.880511 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:04.886081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.886061 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:26:04.886623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.886564 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:26:04.905137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:04.905057 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s"] Nov 01 22:26:05.033307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.033265 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbr6\" (UniqueName: \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.033502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.033391 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.033630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.033607 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.135753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.135713 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.135979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.135960 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbr6\" (UniqueName: \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.136082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.136000 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.136711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.136689 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.138315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.138285 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.148820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.148794 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbr6\" (UniqueName: \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\") pod \"maas-api-revoke-sub-948e632f-db27s\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.202976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.202942 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:05.308545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.307734 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerStarted","Data":"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8"} Nov 01 22:26:05.308545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.307806 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerStarted","Data":"fa4aaeff44fae8a1b9e2d7d7d2fcab0858f613a56fdeb4688639365188749c3b"} Nov 01 22:26:05.342305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.339361 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podStartSLOduration=2.339341794 podStartE2EDuration="2.339341794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:05.336554434 +0000 UTC m=+2519.029534919" watchObservedRunningTime="2026-11-01 22:26:05.339341794 +0000 UTC m=+2519.032322281" Nov 01 22:26:05.566882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:05.565719 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s"] Nov 01 22:26:06.152220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.152184 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:06.158835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.158807 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:06.320563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.320520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" event={"ID":"7cb1ca90-5597-4567-b724-f5e7e5a5cf33","Type":"ContainerStarted","Data":"4b74ba212b8eef563bd8840ee48c1a9721053a58fcf5667f7b6a983eadb03502"} Nov 01 22:26:06.320762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.320595 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" event={"ID":"7cb1ca90-5597-4567-b724-f5e7e5a5cf33","Type":"ContainerStarted","Data":"3f447997e74ab1e5ee0700b8c413d07d16a34fa3a262dc7353a5945a7334f1c2"} Nov 01 22:26:06.321058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.321035 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:06.322196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.322171 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:26:06.342971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:06.342915 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" podStartSLOduration=2.342901234 podStartE2EDuration="2.342901234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:06.338706675 +0000 UTC m=+2520.031687161" watchObservedRunningTime="2026-11-01 22:26:06.342901234 +0000 UTC m=+2520.035881720" Nov 01 22:26:07.331129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:07.331094 5036 generic.go:356] "Generic (PLEG): container finished" podID="7cb1ca90-5597-4567-b724-f5e7e5a5cf33" containerID="4b74ba212b8eef563bd8840ee48c1a9721053a58fcf5667f7b6a983eadb03502" exitCode=0 Nov 01 22:26:07.331558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:07.331161 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" event={"ID":"7cb1ca90-5597-4567-b724-f5e7e5a5cf33","Type":"ContainerDied","Data":"4b74ba212b8eef563bd8840ee48c1a9721053a58fcf5667f7b6a983eadb03502"} Nov 01 22:26:07.550465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:07.548010 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:07.687839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:07.687747 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:08.523655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.523622 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:08.698866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.698773 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\") pod \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " Nov 01 22:26:08.699036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.698972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\" (UniqueName: \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\") pod \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " Nov 01 22:26:08.699257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.699224 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca" pod "7cb1ca90-5597-4567-b724-f5e7e5a5cf33" (UID: "7cb1ca90-5597-4567-b724-f5e7e5a5cf33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:08.699581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.699553 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\") pod \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\" (UID: \"7cb1ca90-5597-4567-b724-f5e7e5a5cf33\") " Nov 01 22:26:08.701236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.701208 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:08.705223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.705186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca" pod "7cb1ca90-5597-4567-b724-f5e7e5a5cf33" (UID: "7cb1ca90-5597-4567-b724-f5e7e5a5cf33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:08.705386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.705362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6" pod "7cb1ca90-5597-4567-b724-f5e7e5a5cf33" (UID: "7cb1ca90-5597-4567-b724-f5e7e5a5cf33"). InnerVolumeSpecName "kube-api-access-9nbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:08.775395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.775360 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:08.803235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.803201 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nbr6\" (UniqueName: \"kubernetes.io/projected/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-kube-api-access-9nbr6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:08.803399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.803238 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb1ca90-5597-4567-b724-f5e7e5a5cf33-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:08.906953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:08.906846 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:09.344522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:09.344488 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:09.353618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:09.353579 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" event={"ID":"7cb1ca90-5597-4567-b724-f5e7e5a5cf33","Type":"ContainerDied","Data":"3f447997e74ab1e5ee0700b8c413d07d16a34fa3a262dc7353a5945a7334f1c2"} Nov 01 22:26:09.354189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:09.354164 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f447997e74ab1e5ee0700b8c413d07d16a34fa3a262dc7353a5945a7334f1c2" Nov 01 22:26:09.354508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:09.354491 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s" Nov 01 22:26:10.553767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:10.553722 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:11.247471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:11.240395 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:11.305939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:11.305896 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:26:11.375731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:11.375700 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:12.483469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:12.470218 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:13.345028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:13.344974 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:14.130227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:14.130190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:14.767826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:14.767777 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:26:14.820682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:14.820551 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:15.120942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:15.120899 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:16.056005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:16.055965 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:17.200037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:17.199992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:17.500131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:17.500055 5036 scope.go:122] "RemoveContainer" containerID="f990e21182c555af2e35480a78f04ecd9f758b04478939fd312ed47de4bf0948" Nov 01 22:26:17.518948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:17.518918 5036 scope.go:122] "RemoveContainer" containerID="5201b43369a1e6a2ebb78c2f1182fcbee083ef087d8796637e61553d88d0894d" Nov 01 22:26:18.036583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:18.036546 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:18.862874 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:18.862835 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:26:19.054145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:19.054111 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:19.773868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:19.773835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:20.521681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:20.521638 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:21.360502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.359541 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:21.832996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.832959 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk"] Nov 01 22:26:21.833825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.833803 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7cb1ca90-5597-4567-b724-f5e7e5a5cf33" containerName="revoke-keys" Nov 01 22:26:21.833892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.833826 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb1ca90-5597-4567-b724-f5e7e5a5cf33" containerName="revoke-keys" Nov 01 22:26:21.833974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.833961 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb1ca90-5597-4567-b724-f5e7e5a5cf33" containerName="revoke-keys" Nov 01 22:26:21.840236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.840213 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:21.845497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.845433 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:26:21.846731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.846676 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:26:21.848869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:21.848833 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk"] Nov 01 22:26:22.022117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.022074 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5lm\" (UniqueName: \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.022289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.022226 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.022537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.022517 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.124457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.124364 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.124626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.124571 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.124696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.124674 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5lm\" (UniqueName: \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.125438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.125406 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.126904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.126883 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.137430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.137398 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5lm\" (UniqueName: \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.169440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.169413 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:22.194092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.194058 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:22.527245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.527208 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk"] Nov 01 22:26:22.527692 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:22.527658 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bd87c1_fde9_4ccc_a58d_fb7f09c4c40b.slice/crio-2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608 WatchSource:0}: Error finding container 2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608: Status 404 returned error can't find the container with id 2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608 Nov 01 22:26:22.671417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.671373 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" event={"ID":"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b","Type":"ContainerStarted","Data":"2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608"} Nov 01 22:26:22.735226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.735192 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5"] Nov 01 22:26:22.748030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.747980 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5"] Nov 01 22:26:22.748211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.748134 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:22.943930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.943894 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:22.944342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.944093 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56rd\" (UniqueName: \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:22.944342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:22.944154 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.045492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.045402 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56rd\" (UniqueName: \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.045673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.045504 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.045761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.045698 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.046364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.046341 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.048077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.048057 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.058177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.058149 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56rd\" (UniqueName: \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\") pod \"maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.088611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.088581 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:23.177843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.177811 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:23.381957 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:23.381925 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7764c1_52a2_48ac_9a54_97416758e0a3.slice/crio-8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea WatchSource:0}: Error finding container 8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea: Status 404 returned error can't find the container with id 8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea Nov 01 22:26:23.382348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.382320 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5"] Nov 01 22:26:23.685345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.685303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" event={"ID":"3e7764c1-52a2-48ac-9a54-97416758e0a3","Type":"ContainerStarted","Data":"8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea"} Nov 01 22:26:23.687716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.687608 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" event={"ID":"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b","Type":"ContainerStarted","Data":"18cef4747ecf9748f9b3f0d0b192ce31a21a6afd85aa7c1bcc0f1353109bd0d9"} Nov 01 22:26:23.708776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:23.708721 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" podStartSLOduration=2.708707385 podStartE2EDuration="2.708707385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:23.705672674 +0000 UTC m=+2537.398653161" watchObservedRunningTime="2026-11-01 22:26:23.708707385 +0000 UTC m=+2537.401687870" Nov 01 22:26:24.041853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.041810 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:24.700160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.700071 5036 generic.go:356] "Generic (PLEG): container finished" podID="c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" containerID="18cef4747ecf9748f9b3f0d0b192ce31a21a6afd85aa7c1bcc0f1353109bd0d9" exitCode=0 Nov 01 22:26:24.700338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.700154 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" event={"ID":"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b","Type":"ContainerDied","Data":"18cef4747ecf9748f9b3f0d0b192ce31a21a6afd85aa7c1bcc0f1353109bd0d9"} Nov 01 22:26:24.702249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.702223 5036 generic.go:356] "Generic (PLEG): container finished" podID="3e7764c1-52a2-48ac-9a54-97416758e0a3" containerID="434acf5a4d7aabe1a04ea720a4246b8ab42097b84da2dea6a00dc20b0caaf6e4" exitCode=0 Nov 01 22:26:24.702386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.702275 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" event={"ID":"3e7764c1-52a2-48ac-9a54-97416758e0a3","Type":"ContainerDied","Data":"434acf5a4d7aabe1a04ea720a4246b8ab42097b84da2dea6a00dc20b0caaf6e4"} Nov 01 22:26:24.832547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:24.832500 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:25.556467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:25.556164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:25.814969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:25.811040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:26.267739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.267707 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:26.353712 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.344360 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:26.419329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.419298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\") pod \"3e7764c1-52a2-48ac-9a54-97416758e0a3\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " Nov 01 22:26:26.419647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.419619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\") pod \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " Nov 01 22:26:26.419893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.419877 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\" (UniqueName: \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\") pod \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " Nov 01 22:26:26.420184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.420160 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\") pod \"3e7764c1-52a2-48ac-9a54-97416758e0a3\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " Nov 01 22:26:26.420487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.420436 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\" (UniqueName: \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\") pod \"3e7764c1-52a2-48ac-9a54-97416758e0a3\" (UID: \"3e7764c1-52a2-48ac-9a54-97416758e0a3\") " Nov 01 22:26:26.420612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.420540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\") pod \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\" (UID: \"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b\") " Nov 01 22:26:26.420612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.420269 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca" pod "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" (UID: "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:26.421305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.421282 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca" pod "3e7764c1-52a2-48ac-9a54-97416758e0a3" (UID: "3e7764c1-52a2-48ac-9a54-97416758e0a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:26.421438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.421297 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.423833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.423801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca" pod "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" (UID: "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:26.430425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.430400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca" pod "3e7764c1-52a2-48ac-9a54-97416758e0a3" (UID: "3e7764c1-52a2-48ac-9a54-97416758e0a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:26.439253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.439178 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm" pod "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" (UID: "c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b"). InnerVolumeSpecName "kube-api-access-gt5lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:26.443363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.443120 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:26.443853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.443833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd" pod "3e7764c1-52a2-48ac-9a54-97416758e0a3" (UID: "3e7764c1-52a2-48ac-9a54-97416758e0a3"). InnerVolumeSpecName "kube-api-access-w56rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:26.529475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.529425 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5lm\" (UniqueName: \"kubernetes.io/projected/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-kube-api-access-gt5lm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.529710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.529696 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.529825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.529813 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56rd\" (UniqueName: \"kubernetes.io/projected/3e7764c1-52a2-48ac-9a54-97416758e0a3-kube-api-access-w56rd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.529908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.529897 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.529990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.529979 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7764c1-52a2-48ac-9a54-97416758e0a3-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:26.577618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.577587 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:26:26.579153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579124 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" containerName="revoke-keys" Nov 01 22:26:26.579378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579358 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" containerName="revoke-keys" Nov 01 22:26:26.579512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579403 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3e7764c1-52a2-48ac-9a54-97416758e0a3" containerName="revoke-keys" Nov 01 22:26:26.579512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579413 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7764c1-52a2-48ac-9a54-97416758e0a3" containerName="revoke-keys" Nov 01 22:26:26.579724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579706 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7764c1-52a2-48ac-9a54-97416758e0a3" containerName="revoke-keys" Nov 01 22:26:26.579791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.579747 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" containerName="revoke-keys" Nov 01 22:26:26.584082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.584058 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.602629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.602595 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" containerID="cri-o://f11f913d5f015f617a71ce57604920dc38ef511ba0523a6b984d2edb7f14be83" gracePeriod=45 Nov 01 22:26:26.609592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.603661 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-521734\"/\"rate-test-model-521734-kserve-self-signed-certs\"" Nov 01 22:26:26.609592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.603927 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:26:26.632975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.632943 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.633151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.633103 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtvs\" (UniqueName: \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.633288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.633270 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.633389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.633375 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.634151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.634127 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.634263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.634222 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.648036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.648005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:26.735438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735403 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735521 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735596 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtvs\" (UniqueName: \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735710 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735784 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735833 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.735933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735918 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.736002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.735979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.736472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.736286 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.791472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.786716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.791472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.787050 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.791472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.787993 5036 generic.go:356] "Generic (PLEG): container finished" podID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerID="f11f913d5f015f617a71ce57604920dc38ef511ba0523a6b984d2edb7f14be83" exitCode=0 Nov 01 22:26:26.791472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.788146 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" event={"ID":"b7ef2efd-927a-43c1-83bf-805f880d0065","Type":"ContainerDied","Data":"f11f913d5f015f617a71ce57604920dc38ef511ba0523a6b984d2edb7f14be83"} Nov 01 22:26:26.823393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.813340 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" event={"ID":"3e7764c1-52a2-48ac-9a54-97416758e0a3","Type":"ContainerDied","Data":"8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea"} Nov 01 22:26:26.823393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.813409 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d852a7c17399d9d316ef904c1795c5d1f087545ee7d3e226a499b931a53e7ea" Nov 01 22:26:26.823393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.813500 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5" Nov 01 22:26:26.840023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.834268 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtvs\" (UniqueName: \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\") pod \"rate-test-model-521734-kserve-84d5785cb5-w5sqp\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:26.888641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.861188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" event={"ID":"c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b","Type":"ContainerDied","Data":"2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608"} Nov 01 22:26:26.888641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.861248 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec30bc634b779bf135924ac3827743848a1858d633a02f048ed7e3eadb84608" Nov 01 22:26:26.888641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.861316 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk" Nov 01 22:26:26.935188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:26.934896 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:27.265266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.265239 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:26:27.343816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.342832 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:26:27.359628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.359592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:27.363239 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:27.363197 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd82296e_8781_49de_8ff3_417e954b022d.slice/crio-bf59a31c79c19e2052e70c4c96b620ebfbb8036d9e3adf0245c213194c0fa0d4 WatchSource:0}: Error finding container bf59a31c79c19e2052e70c4c96b620ebfbb8036d9e3adf0245c213194c0fa0d4: Status 404 returned error can't find the container with id bf59a31c79c19e2052e70c4c96b620ebfbb8036d9e3adf0245c213194c0fa0d4 Nov 01 22:26:27.374508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.374430 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381432 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\" (UniqueName: \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381727 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381772 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.382808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.381828 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\") pod \"b7ef2efd-927a-43c1-83bf-805f880d0065\" (UID: \"b7ef2efd-927a-43c1-83bf-805f880d0065\") " Nov 01 22:26:27.384255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.383695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:27.384255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.384005 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:27.384742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.384708 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:27.387635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.387595 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "kube-api-access-thnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:27.390431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.390391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:27.469714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.463753 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm" pod "b7ef2efd-927a-43c1-83bf-805f880d0065" (UID: "b7ef2efd-927a-43c1-83bf-805f880d0065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488226 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ef2efd-927a-43c1-83bf-805f880d0065-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488273 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thnql\" (UniqueName: \"kubernetes.io/projected/b7ef2efd-927a-43c1-83bf-805f880d0065-kube-api-access-thnql\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488290 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488305 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488319 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.488334 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ef2efd-927a-43c1-83bf-805f880d0065-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:27.509522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.496882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:27.893977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.893934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" event={"ID":"dd82296e-8781-49de-8ff3-417e954b022d","Type":"ContainerStarted","Data":"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9"} Nov 01 22:26:27.894962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.893989 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" event={"ID":"dd82296e-8781-49de-8ff3-417e954b022d","Type":"ContainerStarted","Data":"bf59a31c79c19e2052e70c4c96b620ebfbb8036d9e3adf0245c213194c0fa0d4"} Nov 01 22:26:27.896072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.896045 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" event={"ID":"b7ef2efd-927a-43c1-83bf-805f880d0065","Type":"ContainerDied","Data":"d12afbc5d96e24076f1a7bebb60c8b0d58266a3751fa0940935d10180cc5cef2"} Nov 01 22:26:27.896193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.896089 5036 scope.go:122] "RemoveContainer" containerID="f11f913d5f015f617a71ce57604920dc38ef511ba0523a6b984d2edb7f14be83" Nov 01 22:26:27.896262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.896248 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x" Nov 01 22:26:27.935566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.935537 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:27.937286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.937243 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" podStartSLOduration=1.937230735 podStartE2EDuration="1.937230735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:27.916305433 +0000 UTC m=+2541.609285920" watchObservedRunningTime="2026-11-01 22:26:27.937230735 +0000 UTC m=+2541.630211220" Nov 01 22:26:27.939764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.939741 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:26:27.945002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:27.944979 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/auth-test-model-521734-kserve-78f6f58c8f-5hx7x"] Nov 01 22:26:28.000996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:28.000966 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:28.154529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:28.151152 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:28.428523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:28.428382 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:28.442338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:28.442298 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" path="/var/lib/kubelet/pods/b7ef2efd-927a-43c1-83bf-805f880d0065/volumes" Nov 01 22:26:28.790469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:28.785489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:29.258854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:29.258765 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:29.814698 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:29.814654 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" containerID="cri-o://21c84ec3f6b453215aa32cb8ee36432221d540e0fa2670a3f034107fab2e3794" gracePeriod=45 Nov 01 22:26:29.948121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:29.948090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:30.050464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.043749 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" probeResult="failure" output="Get \"https://10.132.1.44:8000/ready\": dial tcp 10.132.1.44:8000: connect: connection refused" Nov 01 22:26:30.077514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.067288 5036 generic.go:356] "Generic (PLEG): container finished" podID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerID="21c84ec3f6b453215aa32cb8ee36432221d540e0fa2670a3f034107fab2e3794" exitCode=0 Nov 01 22:26:30.077514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.070972 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" event={"ID":"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db","Type":"ContainerDied","Data":"21c84ec3f6b453215aa32cb8ee36432221d540e0fa2670a3f034107fab2e3794"} Nov 01 22:26:30.238440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.238411 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:26:30.406437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406341 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406585 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406624 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406670 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406703 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\" (UniqueName: \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.406949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.406905 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\") pod \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\" (UID: \"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db\") " Nov 01 22:26:30.407667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.407636 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:30.407784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.407739 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.408108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.408057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:30.426714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.426617 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:30.426714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.426674 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "kube-api-access-m66db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:30.443779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.443749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs" pod "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" (UID: "bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:30.510394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.510361 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.510394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.510396 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.510620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.510411 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.510620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.510427 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.510620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.510458 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m66db\" (UniqueName: \"kubernetes.io/projected/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db-kube-api-access-m66db\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:30.651184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:30.651150 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:31.080332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.080288 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" event={"ID":"bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db","Type":"ContainerDied","Data":"dff9b8c42c823f53ff80b8199ce8e570563dc4c6bc045569195d06868cab791d"} Nov 01 22:26:31.080597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.080356 5036 scope.go:122] "RemoveContainer" containerID="21c84ec3f6b453215aa32cb8ee36432221d540e0fa2670a3f034107fab2e3794" Nov 01 22:26:31.080597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.080523 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m" Nov 01 22:26:31.124382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.124349 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:26:31.129689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.129665 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/auth-test-model-d2832c-kserve-586dcf5c49-qjw5m"] Nov 01 22:26:31.622932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.620174 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:31.899669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:31.899553 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:32.033504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.033464 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:32.440818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.440786 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" path="/var/lib/kubelet/pods/bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db/volumes" Nov 01 22:26:32.459380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.459346 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:26:32.460133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460111 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" Nov 01 22:26:32.460232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460134 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" Nov 01 22:26:32.460232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460172 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" Nov 01 22:26:32.460232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460181 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" Nov 01 22:26:32.460394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460380 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfefd7c6-8a4c-46a2-9be5-3e9a3d6349db" containerName="main" Nov 01 22:26:32.460479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.460410 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ef2efd-927a-43c1-83bf-805f880d0065" containerName="main" Nov 01 22:26:32.465954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.465923 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.472847 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.472670 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a00c6a-serving-cert\"" Nov 01 22:26:32.475402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.475365 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:26:32.567420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.567385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.567637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.567458 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.567637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.567624 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.567758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.567748 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9srj\" (UniqueName: \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.581459 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.581407 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:32.669604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.669571 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9srj\" (UniqueName: \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.670095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.669732 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.670095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.669773 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.670095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.669884 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.670685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.670656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.672335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.672306 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.672517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.672495 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.683018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.682988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9srj\" (UniqueName: \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\") pod \"maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:32.731255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.731092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:32.803296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:32.803219 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:33.145127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.144769 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:26:33.191914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.191828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:33.327967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.327928 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:33.465525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.464075 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:33.632685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.632653 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:33.907005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.906494 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:26:33.924936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.924831 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:33.946348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.946324 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:26:33.947363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.946625 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:33.960520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:33.954760 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-56452a-openshift-default-dockercfg-2jkqw\"" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028655 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ktn\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028721 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028780 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028850 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028896 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.028958 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.029063 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.029183 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.045774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.029279 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130340 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ktn\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130395 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130459 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130526 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130574 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130626 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130730 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130856 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.130947 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.139197 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.140190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.139799 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.140500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.140470 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.141031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.141002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.141296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.141273 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.265530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.256199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.312028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.311987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:34.459427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.459193 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.483159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.472979 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ktn\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.483159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.473777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\") pod \"e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:34.546193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.545548 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:26:34.577657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.577597 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" event={"ID":"c89e6f84-ce0b-40cf-a029-e318dc428ee9","Type":"ContainerStarted","Data":"490db5440b9b3dc83f0efbeba24a9c902fc3c294e3ad92bc20c540525ae1edc8"} Nov 01 22:26:34.683413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.683366 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:34.730259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:34.730225 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:35.094909 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:35.094876 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fa88ab_54ba_41e9_870e_47f3f8099334.slice/crio-4b0cfc499035afd43950fb2fe294f0f811694b84773490f72469422e8fb25080 WatchSource:0}: Error finding container 4b0cfc499035afd43950fb2fe294f0f811694b84773490f72469422e8fb25080: Status 404 returned error can't find the container with id 4b0cfc499035afd43950fb2fe294f0f811694b84773490f72469422e8fb25080 Nov 01 22:26:35.096335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.096311 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:26:35.098945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.098902 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:35.099065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.098993 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:35.099125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.099057 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:26:35.589054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.589006 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" event={"ID":"c89e6f84-ce0b-40cf-a029-e318dc428ee9","Type":"ContainerStarted","Data":"c5127b64be6f81374e37938c864c4ec3b894cf1a0446330b91f03f7dc0dc502d"} Nov 01 22:26:35.589614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.589590 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:35.591883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.591798 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerStarted","Data":"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5"} Nov 01 22:26:35.591883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.591849 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerStarted","Data":"4b0cfc499035afd43950fb2fe294f0f811694b84773490f72469422e8fb25080"} Nov 01 22:26:35.611981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.611915 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" podStartSLOduration=3.028519519 podStartE2EDuration="3.611895915s" totalImagesPullingTime="583.376396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:35.609758154 +0000 UTC m=+2549.302738642" watchObservedRunningTime="2026-11-01 22:26:35.611895915 +0000 UTC m=+2549.304876405" Nov 01 22:26:35.640314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.640258 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podStartSLOduration=2.6402433849999998 podStartE2EDuration="2.640243385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:35.634824866 +0000 UTC m=+2549.327805353" watchObservedRunningTime="2026-11-01 22:26:35.640243385 +0000 UTC m=+2549.333223871" Nov 01 22:26:35.650606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.650572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:35.733388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:35.733351 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:36.568469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:36.568142 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:36.740361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:36.740327 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:36.935981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:36.935898 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:36.949056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:36.949034 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:37.516115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.516067 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:37.647318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.647278 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:37.648075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.648051 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:26:37.659705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.659678 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:26:37.822629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.821506 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:37.956842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:37.956795 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:38.627870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:38.627834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:39.091057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:39.091018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:39.330215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:39.295338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:39.761852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:39.761808 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:40.556669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.556628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:40.626036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.625991 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:26:40.626924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.626863 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" podUID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" containerName="maas-api" containerID="cri-o://947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065" gracePeriod=30 Nov 01 22:26:40.658153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.658121 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:26:40.658734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.658706 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerName="maas-api" containerID="cri-o://e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858" gracePeriod=30 Nov 01 22:26:40.748143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.748108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:40.764336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.764308 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk"] Nov 01 22:26:40.767895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.767873 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5a355b-w97qk"] Nov 01 22:26:40.872837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:40.872765 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:41.100007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.045167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.59:8443/health\": dial tcp 10.132.1.59:8443: connect: connection refused" Nov 01 22:26:41.101773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.101701 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:41.264784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.250516 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:41.459468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.457166 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:41.471900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.462484 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5"] Nov 01 22:26:41.471900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.471784 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e25317-rpbp5"] Nov 01 22:26:41.653537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.653408 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:26:41.671934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.670556 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:26:41.755685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.745597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\") pod \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " Nov 01 22:26:41.755685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.745698 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\") pod \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " Nov 01 22:26:41.755685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.745760 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\") pod \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " Nov 01 22:26:41.755685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.745897 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\" (UniqueName: \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\") pod \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\" (UID: \"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c\") " Nov 01 22:26:41.755685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.749103 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca" pod "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" (UID: "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:41.765986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.764857 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs" pod "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" (UID: "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:41.772798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.772759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls" pod "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" (UID: "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:41.773088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.773057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs" pod "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" (UID: "cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c"). InnerVolumeSpecName "kube-api-access-x9mhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:41.835280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.818515 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:41.851927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.849995 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mhs\" (UniqueName: \"kubernetes.io/projected/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-kube-api-access-x9mhs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.851927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.850038 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.851927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.850053 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.851927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.850069 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.854383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.854352 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" containerID="947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065" exitCode=0 Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.854429 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.854497 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" event={"ID":"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c","Type":"ContainerDied","Data":"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065"} Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.854989 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8" event={"ID":"cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c","Type":"ContainerDied","Data":"eca8d1fb9fc64aa7aa8e75b916230eb4c490b719495d66b8451cde1365546b5c"} Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.855025 5036 scope.go:122] "RemoveContainer" containerID="947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065" Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.857304 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.858057 5036 generic.go:356] "Generic (PLEG): container finished" podID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerID="e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858" exitCode=0 Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.858168 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" event={"ID":"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4","Type":"ContainerDied","Data":"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858"} Nov 01 22:26:41.858488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.858209 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" event={"ID":"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4","Type":"ContainerDied","Data":"a5ee1fcafce336ad797b75066dfe38c5f00f7bbb2d42bdd3f2eb345f34a8b9f0"} Nov 01 22:26:41.976527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.976197 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\") pod \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " Nov 01 22:26:41.976527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.976418 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\" (UniqueName: \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\") pod \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " Nov 01 22:26:41.976527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.976480 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\") pod \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " Nov 01 22:26:41.976854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:41.976601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") pod \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\" (UID: \"8a8d4eb0-7f6a-4827-9178-176f2cd3eae4\") " Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.057702 5036 scope.go:122] "RemoveContainer" containerID="947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065" Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.057697 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca" pod "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" (UID: "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.057864 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6" pod "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" (UID: "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4"). InnerVolumeSpecName "kube-api-access-5xmk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.058472 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs" pod "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" (UID: "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.058711 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls" pod "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" (UID: "8a8d4eb0-7f6a-4827-9178-176f2cd3eae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:42.059826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.058723 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:42.060342 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:42.060222 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065\": container with ID starting with 947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065 not found: ID does not exist" containerID="947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065" Nov 01 22:26:42.063585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.060264 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065"} err="failed to get container status \"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065\": rpc error: code = NotFound desc = could not find container \"947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065\": container with ID starting with 947473a0dfeb01b1cc5ad4a2557aec9361d221650392efb8da41dfe495e81065 not found: ID does not exist" Nov 01 22:26:42.063585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.060481 5036 scope.go:122] "RemoveContainer" containerID="e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858" Nov 01 22:26:42.074283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.074254 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:26:42.081154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.080916 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmk6\" (UniqueName: \"kubernetes.io/projected/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-kube-api-access-5xmk6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.081154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.080947 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.081154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.080963 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.081154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.080978 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.084587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.084534 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a355b-957644c45-p2vb8"] Nov 01 22:26:42.091233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.091137 5036 scope.go:122] "RemoveContainer" containerID="e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858" Nov 01 22:26:42.091672 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:42.091648 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858\": container with ID starting with e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858 not found: ID does not exist" containerID="e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858" Nov 01 22:26:42.091771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.091682 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858"} err="failed to get container status \"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858\": rpc error: code = NotFound desc = could not find container \"e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858\": container with ID starting with e9a8d65da0daa3625a85283df34cfcbfe9700c384315a72236047cfb9fb1a858 not found: ID does not exist" Nov 01 22:26:42.390800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.390715 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" containerID="cri-o://c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8" gracePeriod=45 Nov 01 22:26:42.440881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.440851 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7764c1-52a2-48ac-9a54-97416758e0a3" path="/var/lib/kubelet/pods/3e7764c1-52a2-48ac-9a54-97416758e0a3/volumes" Nov 01 22:26:42.441413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.441396 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b" path="/var/lib/kubelet/pods/c8bd87c1-fde9-4ccc-a58d-fb7f09c4c40b/volumes" Nov 01 22:26:42.441901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.441886 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" path="/var/lib/kubelet/pods/cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c/volumes" Nov 01 22:26:42.465759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.465729 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.45:8000/ready\": dial tcp 10.132.1.45:8000: connect: connection refused" Nov 01 22:26:42.543539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.540690 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:42.800015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.799985 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:26:42.870894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.870861 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk" Nov 01 22:26:42.872638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.872606 5036 generic.go:356] "Generic (PLEG): container finished" podID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerID="c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8" exitCode=0 Nov 01 22:26:42.872790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.872648 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" Nov 01 22:26:42.872790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.872691 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" event={"ID":"3be6be11-01b8-414b-9c36-1c1ab2a4a17b","Type":"ContainerDied","Data":"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8"} Nov 01 22:26:42.872790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.872730 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t" event={"ID":"3be6be11-01b8-414b-9c36-1c1ab2a4a17b","Type":"ContainerDied","Data":"427fc305596ac0c12e822bfc36bb99dccfcb6b41a24c982bab5c715d56cd558f"} Nov 01 22:26:42.872790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.872756 5036 scope.go:122] "RemoveContainer" containerID="c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8" Nov 01 22:26:42.884942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.884923 5036 scope.go:122] "RemoveContainer" containerID="c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8" Nov 01 22:26:42.885261 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:42.885226 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8\": container with ID starting with c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8 not found: ID does not exist" containerID="c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8" Nov 01 22:26:42.885379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.885271 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8"} err="failed to get container status \"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8\": rpc error: code = NotFound desc = could not find container \"c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8\": container with ID starting with c9f2e7cc2a22078ce5417b9afd88d20c3e6cbe53f285fa25b75463047c69f2f8 not found: ID does not exist" Nov 01 22:26:42.894983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.894491 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:26:42.896859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.896837 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\" (UniqueName: \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.896992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.896930 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.897142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897125 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.897212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.897273 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897261 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.897396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897380 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\") pod \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\" (UID: \"3be6be11-01b8-414b-9c36-1c1ab2a4a17b\") " Nov 01 22:26:42.897571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897539 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:42.897770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.897749 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:42.898142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.898113 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:42.898252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.898186 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.898314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.898267 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:42.899990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.899964 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "kube-api-access-8lkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:42.900671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.900646 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:42.900786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.900768 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs" pod "3be6be11-01b8-414b-9c36-1c1ab2a4a17b" (UID: "3be6be11-01b8-414b-9c36-1c1ab2a4a17b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:42.901458 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:42.901423 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e25317-86c87c86f5-lgpkk"] Nov 01 22:26:43.000194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.000157 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:43.000194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.000192 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:43.000194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.000202 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:43.000434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.000211 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkrh\" (UniqueName: \"kubernetes.io/projected/3be6be11-01b8-414b-9c36-1c1ab2a4a17b-kube-api-access-8lkrh\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:43.204166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.202197 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:26:43.207348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.207313 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-2c5135/rate-test-model-2c5135-kserve-5d5b597ddf-zw49t"] Nov 01 22:26:43.363012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:43.362983 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:44.157916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:44.157867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:44.340717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:44.340679 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:26:44.442713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:44.442638 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" path="/var/lib/kubelet/pods/3be6be11-01b8-414b-9c36-1c1ab2a4a17b/volumes" Nov 01 22:26:44.443124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:44.443108 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" path="/var/lib/kubelet/pods/8a8d4eb0-7f6a-4827-9178-176f2cd3eae4/volumes" Nov 01 22:26:45.552618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:45.550830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:46.614198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:46.614147 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:47.438806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:47.438711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.087483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.087434 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.242180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.242140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.460765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.460670 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.720307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.720216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.864751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.864713 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:48.884374 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:48.883946 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:26:48.951465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.951058 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn"] Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952241 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" containerName="maas-api" Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952270 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" containerName="maas-api" Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952284 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerName="maas-api" Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952293 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerName="maas-api" Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952309 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" Nov 01 22:26:48.952507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952317 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" Nov 01 22:26:48.952951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952553 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5e2c9e-1db6-49c6-ba77-2ac9cf3edb4c" containerName="maas-api" Nov 01 22:26:48.952951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952568 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8d4eb0-7f6a-4827-9178-176f2cd3eae4" containerName="maas-api" Nov 01 22:26:48.952951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.952581 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be6be11-01b8-414b-9c36-1c1ab2a4a17b" containerName="main" Nov 01 22:26:48.958746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.958720 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:48.974728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.974661 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn"] Nov 01 22:26:48.982962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.982939 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:26:48.983502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.983406 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" containerID="cri-o://b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018" gracePeriod=30 Nov 01 22:26:48.995151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.993528 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:26:48.995151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:48.993767 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" containerID="cri-o://56056fac195946d8b1e19875a984f1226432cf0f1599d83dd84f2dcacf21a93a" gracePeriod=30 Nov 01 22:26:49.007570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.007544 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:26:49.007692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.007644 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:26:49.029208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.029179 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:49.059247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.059215 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5nk\" (UniqueName: \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.059395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.059262 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.059395 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.059299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.186469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.175696 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5nk\" (UniqueName: \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.186469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.175750 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.186469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.175790 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.212471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.204290 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.212471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.210974 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.264639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.264551 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5nk\" (UniqueName: \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.336695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.336665 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:49.409668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.409502 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:49.467305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.467261 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p"] Nov 01 22:26:49.521755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.475674 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.552348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.544226 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p"] Nov 01 22:26:49.634503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.633883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.634503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.634089 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff95b\" (UniqueName: \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.634503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.634182 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.735117 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:49.735085 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccd4423_3e02_492b_adb0_601c1bb08851.slice/crio-454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665 WatchSource:0}: Error finding container 454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665: Status 404 returned error can't find the container with id 454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665 Nov 01 22:26:49.735770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.735741 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.736023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.735991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.736179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.736152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff95b\" (UniqueName: \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.736722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.736694 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.738558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.738537 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.739082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.739055 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn"] Nov 01 22:26:49.748163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.748132 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff95b\" (UniqueName: \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\") pod \"maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:49.843208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:49.843179 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:50.133600 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:50.133568 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cde95b_ec5a_41e3_bcd5_d407334a6817.slice/crio-7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75 WatchSource:0}: Error finding container 7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75: Status 404 returned error can't find the container with id 7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75 Nov 01 22:26:50.134298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.134268 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p"] Nov 01 22:26:50.281500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.277985 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:50.389542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.388651 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" event={"ID":"7ccd4423-3e02-492b-adb0-601c1bb08851","Type":"ContainerStarted","Data":"fae740b97479e540e03b9db903afc5c6249a2781ea3d36ccbc03cd852ba94f96"} Nov 01 22:26:50.389542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.388721 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" event={"ID":"7ccd4423-3e02-492b-adb0-601c1bb08851","Type":"ContainerStarted","Data":"454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665"} Nov 01 22:26:50.392407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.392376 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" event={"ID":"10cde95b-ec5a-41e3-bcd5-d407334a6817","Type":"ContainerStarted","Data":"7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75"} Nov 01 22:26:50.411253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:50.411197 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" podStartSLOduration=2.411179924 podStartE2EDuration="2.411179924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:50.409092441 +0000 UTC m=+2564.102072928" watchObservedRunningTime="2026-11-01 22:26:50.411179924 +0000 UTC m=+2564.104160411" Nov 01 22:26:51.031088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.031003 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:51.404642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.404556 5036 generic.go:356] "Generic (PLEG): container finished" podID="10cde95b-ec5a-41e3-bcd5-d407334a6817" containerID="eb7c7aa75ed2b5a376975064104e569e3dd1b430ae62c07f4951485298154bac" exitCode=0 Nov 01 22:26:51.405050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.404645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" event={"ID":"10cde95b-ec5a-41e3-bcd5-d407334a6817","Type":"ContainerDied","Data":"eb7c7aa75ed2b5a376975064104e569e3dd1b430ae62c07f4951485298154bac"} Nov 01 22:26:51.406427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.406403 5036 generic.go:356] "Generic (PLEG): container finished" podID="7ccd4423-3e02-492b-adb0-601c1bb08851" containerID="fae740b97479e540e03b9db903afc5c6249a2781ea3d36ccbc03cd852ba94f96" exitCode=0 Nov 01 22:26:51.406599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.406499 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" event={"ID":"7ccd4423-3e02-492b-adb0-601c1bb08851","Type":"ContainerDied","Data":"fae740b97479e540e03b9db903afc5c6249a2781ea3d36ccbc03cd852ba94f96"} Nov 01 22:26:51.699733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:51.699652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:52.539471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.536392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:52.722101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.717626 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:52.736827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.732193 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:52.742536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.742508 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:26:52.743788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.743758 5036 cpu_manager.go:414] "containerMap: removing container" podUID="10cde95b-ec5a-41e3-bcd5-d407334a6817" containerName="revoke-keys" Nov 01 22:26:52.743915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.743789 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cde95b-ec5a-41e3-bcd5-d407334a6817" containerName="revoke-keys" Nov 01 22:26:52.743988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.743960 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cde95b-ec5a-41e3-bcd5-d407334a6817" containerName="revoke-keys" Nov 01 22:26:52.752265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.752238 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.757809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.757780 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:26:52.768089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.768045 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"default-dockercfg-zkjp8\"" Nov 01 22:26:52.768528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.768138 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"rate-test-model-d2832c-kserve-self-signed-certs\"" Nov 01 22:26:52.768528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.768199 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"kube-root-ca.crt\"" Nov 01 22:26:52.768528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.768284 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d2832c\"/\"openshift-service-ca.crt\"" Nov 01 22:26:52.832468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.832300 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.947667 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\" (UniqueName: \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\") pod \"10cde95b-ec5a-41e3-bcd5-d407334a6817\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.947788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\") pod \"10cde95b-ec5a-41e3-bcd5-d407334a6817\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.948098 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\") pod \"10cde95b-ec5a-41e3-bcd5-d407334a6817\" (UID: \"10cde95b-ec5a-41e3-bcd5-d407334a6817\") " Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.956851 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca" pod "10cde95b-ec5a-41e3-bcd5-d407334a6817" (UID: "10cde95b-ec5a-41e3-bcd5-d407334a6817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.966141 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.966253 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.975665 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.975817 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzq8\" (UniqueName: \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.975918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.976083 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:52.992524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:52.976282 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.096470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.096199 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\") pod \"7ccd4423-3e02-492b-adb0-601c1bb08851\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " Nov 01 22:26:53.096470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.096399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\") pod \"7ccd4423-3e02-492b-adb0-601c1bb08851\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " Nov 01 22:26:53.096741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.096567 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\" (UniqueName: \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\") pod \"7ccd4423-3e02-492b-adb0-601c1bb08851\" (UID: \"7ccd4423-3e02-492b-adb0-601c1bb08851\") " Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.096968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzq8\" (UniqueName: \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097055 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097190 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097302 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097379 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097459 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.097988 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.099154 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca" pod "7ccd4423-3e02-492b-adb0-601c1bb08851" (UID: "7ccd4423-3e02-492b-adb0-601c1bb08851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.101595 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.106778 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca" pod "7ccd4423-3e02-492b-adb0-601c1bb08851" (UID: "7ccd4423-3e02-492b-adb0-601c1bb08851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:53.109246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.107541 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.111666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.111633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b" pod "10cde95b-ec5a-41e3-bcd5-d407334a6817" (UID: "10cde95b-ec5a-41e3-bcd5-d407334a6817"). InnerVolumeSpecName "kube-api-access-ff95b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:53.111976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.111949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk" pod "7ccd4423-3e02-492b-adb0-601c1bb08851" (UID: "7ccd4423-3e02-492b-adb0-601c1bb08851"). InnerVolumeSpecName "kube-api-access-lr5nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:53.117168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.117139 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.132752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.132718 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca" pod "10cde95b-ec5a-41e3-bcd5-d407334a6817" (UID: "10cde95b-ec5a-41e3-bcd5-d407334a6817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:26:53.133343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.133313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.137649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.137016 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzq8\" (UniqueName: \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\") pod \"rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.137649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.137145 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:53.200029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.199970 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.200247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.200229 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd4423-3e02-492b-adb0-601c1bb08851-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.200368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.200355 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr5nk\" (UniqueName: \"kubernetes.io/projected/7ccd4423-3e02-492b-adb0-601c1bb08851-kube-api-access-lr5nk\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.200478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.200463 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff95b\" (UniqueName: \"kubernetes.io/projected/10cde95b-ec5a-41e3-bcd5-d407334a6817-kube-api-access-ff95b\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.200592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.200580 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cde95b-ec5a-41e3-bcd5-d407334a6817-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.341118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.340915 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:53.448665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.448630 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" Nov 01 22:26:53.448863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.448739 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn" event={"ID":"7ccd4423-3e02-492b-adb0-601c1bb08851","Type":"ContainerDied","Data":"454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665"} Nov 01 22:26:53.448863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.448802 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454054b53a6755050e6906e1651a51aebbb1241439b5b6bcee65646e7cdee665" Nov 01 22:26:53.450611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.450587 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" event={"ID":"10cde95b-ec5a-41e3-bcd5-d407334a6817","Type":"ContainerDied","Data":"7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75"} Nov 01 22:26:53.450775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.450628 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b49aede101a4f15280c51b4447bb829f6f73286d1acf948096903edc3af3d75" Nov 01 22:26:53.450775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.450681 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p" Nov 01 22:26:53.481992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:53.481723 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:26:53.483541 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:26:53.483518 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c43457f_3c85_4291_b7fc_6e7918fbfbf1.slice/crio-73f1bdb5be94e35a48ee20e9a4beb1550f5032eb7733cd178aa76964647c69cb WatchSource:0}: Error finding container 73f1bdb5be94e35a48ee20e9a4beb1550f5032eb7733cd178aa76964647c69cb: Status 404 returned error can't find the container with id 73f1bdb5be94e35a48ee20e9a4beb1550f5032eb7733cd178aa76964647c69cb Nov 01 22:26:54.192620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.188664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:54.432742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.432255 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:26:54.461916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.461878 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerID="b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018" exitCode=0 Nov 01 22:26:54.462109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.461927 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" Nov 01 22:26:54.462109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.461977 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" event={"ID":"6f0e0635-0e28-4f51-8f97-349b69c0f3f8","Type":"ContainerDied","Data":"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018"} Nov 01 22:26:54.462109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.462029 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd" event={"ID":"6f0e0635-0e28-4f51-8f97-349b69c0f3f8","Type":"ContainerDied","Data":"7093f0969d98c54fcf5ae74426f383b3553fa307d1f5267526893d63329dbefe"} Nov 01 22:26:54.462109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.462040 5036 scope.go:122] "RemoveContainer" containerID="b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018" Nov 01 22:26:54.464398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.464370 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" event={"ID":"0c43457f-3c85-4291-b7fc-6e7918fbfbf1","Type":"ContainerStarted","Data":"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1"} Nov 01 22:26:54.464578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.464408 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" event={"ID":"0c43457f-3c85-4291-b7fc-6e7918fbfbf1","Type":"ContainerStarted","Data":"73f1bdb5be94e35a48ee20e9a4beb1550f5032eb7733cd178aa76964647c69cb"} Nov 01 22:26:54.466543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.466521 5036 generic.go:356] "Generic (PLEG): container finished" podID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerID="56056fac195946d8b1e19875a984f1226432cf0f1599d83dd84f2dcacf21a93a" exitCode=0 Nov 01 22:26:54.466666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.466560 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" event={"ID":"cc28e919-b89c-4a2e-b221-2cf7ffc73448","Type":"ContainerDied","Data":"56056fac195946d8b1e19875a984f1226432cf0f1599d83dd84f2dcacf21a93a"} Nov 01 22:26:54.478255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.478225 5036 scope.go:122] "RemoveContainer" containerID="b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018" Nov 01 22:26:54.478676 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:26:54.478643 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018\": container with ID starting with b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018 not found: ID does not exist" containerID="b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018" Nov 01 22:26:54.478773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.478688 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018"} err="failed to get container status \"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018\": rpc error: code = NotFound desc = could not find container \"b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018\": container with ID starting with b54d3a3bed83faf54dd032f455cb199dfda124ec2c04a981c24cb3caa229a018 not found: ID does not exist" Nov 01 22:26:54.480117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.479864 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:26:54.525268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.525209 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" podStartSLOduration=2.525190613 podStartE2EDuration="2.525190613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:54.5225029 +0000 UTC m=+2568.215483387" watchObservedRunningTime="2026-11-01 22:26:54.525190613 +0000 UTC m=+2568.218171103" Nov 01 22:26:54.539793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.539763 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.539916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.539844 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.539916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.539888 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.539941 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.539977 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540011 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540073 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.540282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540188 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540261 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540342 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540404 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540489 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540530 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540654 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540659 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.540759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540723 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.540940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540858 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\") pod \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\" (UID: \"cc28e919-b89c-4a2e-b221-2cf7ffc73448\") " Nov 01 22:26:54.540940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540895 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.541048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.540962 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\") pod \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\" (UID: \"6f0e0635-0e28-4f51-8f97-349b69c0f3f8\") " Nov 01 22:26:54.541885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.541860 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.541990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.541892 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.541990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.541908 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.547013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.545978 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.547013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.546167 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.547417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.547397 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.548509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.548429 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.548719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.548551 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "kube-api-access-gf7dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.548785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.548685 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.548871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.548852 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:54.549952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.549917 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:54.550184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.550165 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.550360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.550336 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.550627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.550600 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo" pod "6f0e0635-0e28-4f51-8f97-349b69c0f3f8" (UID: "6f0e0635-0e28-4f51-8f97-349b69c0f3f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:54.550709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.550671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.554651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.554582 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:54.554651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.554626 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.555119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.555101 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7" pod "cc28e919-b89c-4a2e-b221-2cf7ffc73448" (UID: "cc28e919-b89c-4a2e-b221-2cf7ffc73448"). InnerVolumeSpecName "kube-api-access-lkhv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.643928 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.643962 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.643976 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.643989 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhv7\" (UniqueName: \"kubernetes.io/projected/cc28e919-b89c-4a2e-b221-2cf7ffc73448-kube-api-access-lkhv7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644004 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644017 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644030 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644041 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644053 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644064 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644079 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644091 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7dm\" (UniqueName: \"kubernetes.io/projected/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-kube-api-access-gf7dm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644104 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644116 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28e919-b89c-4a2e-b221-2cf7ffc73448-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.644361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.644128 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0e0635-0e28-4f51-8f97-349b69c0f3f8-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.809473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.797494 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:26:54.809473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:54.802996 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-h7rfd"] Nov 01 22:26:55.137739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.137708 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:26:55.147919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.147881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:55.476528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.476410 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" Nov 01 22:26:55.476528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.476438 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts" event={"ID":"cc28e919-b89c-4a2e-b221-2cf7ffc73448","Type":"ContainerDied","Data":"da687c11d72da60b660db4114546b6d770dd9a4a3dc2f57364f65b455e393ca7"} Nov 01 22:26:55.477064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.476533 5036 scope.go:122] "RemoveContainer" containerID="56056fac195946d8b1e19875a984f1226432cf0f1599d83dd84f2dcacf21a93a" Nov 01 22:26:55.523159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.523134 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:26:55.528645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.528617 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-vpfts"] Nov 01 22:26:55.944823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.944790 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:26:55.945575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945539 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" Nov 01 22:26:55.945575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945565 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945600 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7ccd4423-3e02-492b-adb0-601c1bb08851" containerName="revoke-keys" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945610 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccd4423-3e02-492b-adb0-601c1bb08851" containerName="revoke-keys" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945643 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945652 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945861 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" containerName="istio-proxy" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945886 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" containerName="istio-proxy" Nov 01 22:26:55.945935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.945899 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccd4423-3e02-492b-adb0-601c1bb08851" containerName="revoke-keys" Nov 01 22:26:55.952146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.952122 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:55.963327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.963299 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:26:55.967258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:55.966752 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2a68ac-serving-cert\"" Nov 01 22:26:56.021593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.021550 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:56.113474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.103854 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.113474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.103934 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.113474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.104031 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.113474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.104099 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67c2w\" (UniqueName: \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.152878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.152834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:56.206422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.206331 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.206422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.206404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.206682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.206517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.206682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.206579 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67c2w\" (UniqueName: \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.207223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.207192 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.208943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.208920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.209110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.209086 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.218882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.218858 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67c2w\" (UniqueName: \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\") pod \"maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.280648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.280613 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:56.313853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.313821 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:56.463570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.463077 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0e0635-0e28-4f51-8f97-349b69c0f3f8" path="/var/lib/kubelet/pods/6f0e0635-0e28-4f51-8f97-349b69c0f3f8/volumes" Nov 01 22:26:56.464470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.463915 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc28e919-b89c-4a2e-b221-2cf7ffc73448" path="/var/lib/kubelet/pods/cc28e919-b89c-4a2e-b221-2cf7ffc73448/volumes" Nov 01 22:26:56.744477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.744379 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:26:56.914037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:56.914002 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:57.578800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:57.578764 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:57.774373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:57.774338 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" event={"ID":"eec00664-bafb-4423-9c33-e154b9a4c5ba","Type":"ContainerStarted","Data":"e7d8caca4d22d428a16fa0fcb4150dbe95368872b42280ae8cf0ddaaf3383c8a"} Nov 01 22:26:57.774826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:57.774396 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" event={"ID":"eec00664-bafb-4423-9c33-e154b9a4c5ba","Type":"ContainerStarted","Data":"32173950f126d7221493f502b79e2d442c4ad38d85f485183623d4bfd168d678"} Nov 01 22:26:57.774826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:57.774429 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:26:57.799968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:57.799917 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" podStartSLOduration=2.343790889 podStartE2EDuration="2.799902582s" totalImagesPullingTime="456.111693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:57.794642838 +0000 UTC m=+2571.487623326" watchObservedRunningTime="2026-11-01 22:26:57.799902582 +0000 UTC m=+2571.492883065" Nov 01 22:26:58.534297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:58.534259 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:59.436693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:26:59.436641 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:00.359480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:00.359430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:00.800875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:00.800828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:01.221469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:01.221357 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:01.949182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:01.949150 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:27:01.955015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:01.954987 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:01.964367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:01.964337 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-56452a-serving-cert\"" Nov 01 22:27:01.979937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:01.979910 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:27:02.028958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.028914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:02.102304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.102269 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.102522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.102359 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.102599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.102521 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csh2l\" (UniqueName: \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.102714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.102688 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.210519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.209638 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.210519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.209774 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csh2l\" (UniqueName: \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.210519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.209914 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.210519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.210150 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.225468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.218178 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:02.225468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.219074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.247466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.242228 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.247466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.243859 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.254800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.253238 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csh2l\" (UniqueName: \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\") pod \"maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.289546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.289512 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:02.593717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.593683 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:27:02.594212 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:02.594179 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d621b6_2c69_4c46_b774_b9c8027d4a7d.slice/crio-80f4dceedc9bd28801c71d8e0c253369f11325a8edd68586ef3ad6c078244aef WatchSource:0}: Error finding container 80f4dceedc9bd28801c71d8e0c253369f11325a8edd68586ef3ad6c078244aef: Status 404 returned error can't find the container with id 80f4dceedc9bd28801c71d8e0c253369f11325a8edd68586ef3ad6c078244aef Nov 01 22:27:02.859557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:02.852848 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" event={"ID":"72d621b6-2c69-4c46-b774-b9c8027d4a7d","Type":"ContainerStarted","Data":"80f4dceedc9bd28801c71d8e0c253369f11325a8edd68586ef3ad6c078244aef"} Nov 01 22:27:03.018212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.018174 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:03.036675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.033493 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc"] Nov 01 22:27:03.050465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.048537 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2"] Nov 01 22:27:03.073134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.069995 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19aa3a2-xb5zc"] Nov 01 22:27:03.091691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.091613 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de1f199-tcvk2"] Nov 01 22:27:03.138593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.138507 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:27:03.153855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.153829 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:27:03.569938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.568642 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:03.712466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.703906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:03.820520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.819611 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:27:03.838557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.837564 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:03.949891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.949820 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" event={"ID":"72d621b6-2c69-4c46-b774-b9c8027d4a7d","Type":"ContainerStarted","Data":"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e"} Nov 01 22:27:03.950797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:03.950773 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:04.045933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.035321 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" podStartSLOduration=2.42400412 podStartE2EDuration="3.03529929s" totalImagesPullingTime="611.29517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:04.010413603 +0000 UTC m=+2577.703394095" watchObservedRunningTime="2026-11-01 22:27:04.03529929 +0000 UTC m=+2577.728279777" Nov 01 22:27:04.046925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.045813 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:04.048344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.048322 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:27:04.441124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.441092 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b974f6-4023-4510-a5ee-72f3298c112e" path="/var/lib/kubelet/pods/38b974f6-4023-4510-a5ee-72f3298c112e/volumes" Nov 01 22:27:04.441602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.441585 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9586f0-1545-4678-ada0-550f745b70c0" path="/var/lib/kubelet/pods/5d9586f0-1545-4678-ada0-550f745b70c0/volumes" Nov 01 22:27:04.725614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:04.725519 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:05.464778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:05.464740 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:05.961182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:05.961143 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:27:05.961406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:05.961380 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" podUID="9173fec5-0a69-4315-93dd-3791c90ce8be" containerName="maas-api" containerID="cri-o://a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723" gracePeriod=30 Nov 01 22:27:06.044325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.044293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:06.169924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.169886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:06.316472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.316196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:06.720183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.663350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:06.791331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.791299 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p"] Nov 01 22:27:06.838762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.838719 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:06.847762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.847724 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-829cb0-p2g7p"] Nov 01 22:27:06.884593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.884566 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:27:06.968149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:06.968122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.042025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.041992 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\") pod \"9173fec5-0a69-4315-93dd-3791c90ce8be\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " Nov 01 22:27:07.042257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.042035 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\") pod \"9173fec5-0a69-4315-93dd-3791c90ce8be\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " Nov 01 22:27:07.042695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.042208 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") pod \"9173fec5-0a69-4315-93dd-3791c90ce8be\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " Nov 01 22:27:07.042695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.042385 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca" pod "9173fec5-0a69-4315-93dd-3791c90ce8be" (UID: "9173fec5-0a69-4315-93dd-3791c90ce8be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:07.043485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.043202 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\" (UniqueName: \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\") pod \"9173fec5-0a69-4315-93dd-3791c90ce8be\" (UID: \"9173fec5-0a69-4315-93dd-3791c90ce8be\") " Nov 01 22:27:07.044260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.044229 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9173fec5-0a69-4315-93dd-3791c90ce8be-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.046530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.045480 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls" pod "9173fec5-0a69-4315-93dd-3791c90ce8be" (UID: "9173fec5-0a69-4315-93dd-3791c90ce8be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:07.046530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.045874 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs" pod "9173fec5-0a69-4315-93dd-3791c90ce8be" (UID: "9173fec5-0a69-4315-93dd-3791c90ce8be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:07.046530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.046420 5036 generic.go:356] "Generic (PLEG): container finished" podID="9173fec5-0a69-4315-93dd-3791c90ce8be" containerID="a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723" exitCode=0 Nov 01 22:27:07.046781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.046553 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" event={"ID":"9173fec5-0a69-4315-93dd-3791c90ce8be","Type":"ContainerDied","Data":"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723"} Nov 01 22:27:07.046781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.046622 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" event={"ID":"9173fec5-0a69-4315-93dd-3791c90ce8be","Type":"ContainerDied","Data":"f819b7396a3f26d4209c47ee83bd2b22135df559156afdd37b52a0dd0981d2e1"} Nov 01 22:27:07.046781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.046647 5036 scope.go:122] "RemoveContainer" containerID="a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723" Nov 01 22:27:07.046781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.046568 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg" Nov 01 22:27:07.050607 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.050582 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9" pod "9173fec5-0a69-4315-93dd-3791c90ce8be" (UID: "9173fec5-0a69-4315-93dd-3791c90ce8be"). InnerVolumeSpecName "kube-api-access-f2lr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:07.079748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.079727 5036 scope.go:122] "RemoveContainer" containerID="a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723" Nov 01 22:27:07.080085 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:07.080060 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723\": container with ID starting with a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723 not found: ID does not exist" containerID="a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723" Nov 01 22:27:07.080161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.080096 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723"} err="failed to get container status \"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723\": rpc error: code = NotFound desc = could not find container \"a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723\": container with ID starting with a689141bfad1095f38a07c3aa6fc52e820878fc599bec93b6c5c195c8bfb3723 not found: ID does not exist" Nov 01 22:27:07.152336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.152305 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.152336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.152334 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9173fec5-0a69-4315-93dd-3791c90ce8be-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.152602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.152346 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2lr9\" (UniqueName: \"kubernetes.io/projected/9173fec5-0a69-4315-93dd-3791c90ce8be-kube-api-access-f2lr9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.388933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.388899 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:27:07.407520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.407489 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-829cb0-6556cf55c5-spnxg"] Nov 01 22:27:07.411117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.411090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.544669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.544397 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.689589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.689486 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.823471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.821068 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.903469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.901480 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn"] Nov 01 22:27:07.978857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.970009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:07.996179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:07.988223 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad047b-rmwgn"] Nov 01 22:27:08.233181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:08.232909 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:08.442393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:08.442354 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cde95b-ec5a-41e3-bcd5-d407334a6817" path="/var/lib/kubelet/pods/10cde95b-ec5a-41e3-bcd5-d407334a6817/volumes" Nov 01 22:27:08.442902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:08.442885 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccd4423-3e02-492b-adb0-601c1bb08851" path="/var/lib/kubelet/pods/7ccd4423-3e02-492b-adb0-601c1bb08851/volumes" Nov 01 22:27:08.443334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:08.443320 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9173fec5-0a69-4315-93dd-3791c90ce8be" path="/var/lib/kubelet/pods/9173fec5-0a69-4315-93dd-3791c90ce8be/volumes" Nov 01 22:27:09.131000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:09.130895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:09.796960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:09.796930 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:10.530397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:10.530363 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:11.035111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:11.035081 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:27:11.238129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:11.238090 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:11.939994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:11.939957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:12.654427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:12.654388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:13.027124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:13.027089 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n"] Nov 01 22:27:13.031765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:13.031738 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba3aecf-bc74n"] Nov 01 22:27:13.429434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:13.429355 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.027658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.027619 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8"] Nov 01 22:27:14.031887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.031856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.033005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.032985 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d7f998f-766w8"] Nov 01 22:27:14.157154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.157116 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.378365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.328062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.399487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.395294 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:27:14.399487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.395573 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" containerID="cri-o://df631f6682a0c274fc38cf510beb860a574cdab8164570b6c253b72872f78fd8" gracePeriod=30 Nov 01 22:27:14.713271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.713115 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.880467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.878640 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a8be8f-015e-4152-8885-81b244e35b78" path="/var/lib/kubelet/pods/11a8be8f-015e-4152-8885-81b244e35b78/volumes" Nov 01 22:27:14.880467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.879481 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694" path="/var/lib/kubelet/pods/ae7d8e28-6f61-4a9b-a73c-cf7ab2d32694/volumes" Nov 01 22:27:14.880467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.880019 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz"] Nov 01 22:27:14.885836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.881088 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9173fec5-0a69-4315-93dd-3791c90ce8be" containerName="maas-api" Nov 01 22:27:14.886034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.886015 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9173fec5-0a69-4315-93dd-3791c90ce8be" containerName="maas-api" Nov 01 22:27:14.886952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.886427 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9173fec5-0a69-4315-93dd-3791c90ce8be" containerName="maas-api" Nov 01 22:27:14.902538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.898300 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz"] Nov 01 22:27:14.902538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.898415 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:14.902538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.902339 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:27:14.903086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.903030 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:27:14.948463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.947825 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.981539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.981409 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:14.982028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.981915 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:14.982267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:14.982244 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlsx\" (UniqueName: \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.033006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.032973 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j"] Nov 01 22:27:15.039957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.039930 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beda212e-sqm9j"] Nov 01 22:27:15.084274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.084243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:15.085381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.085352 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.085500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.085458 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.085565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.085514 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlsx\" (UniqueName: \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.086128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.086104 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.088005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.087977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.097900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.097874 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlsx\" (UniqueName: \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.213182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.213143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:15.235739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.235660 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:15.365467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.361868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:15.648071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.648038 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl"] Nov 01 22:27:15.663132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.662800 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.676170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.676138 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl"] Nov 01 22:27:15.735369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.735337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6f9\" (UniqueName: \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.735754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.735722 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.735858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.735779 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.737778 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:15.737756 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb143f291_d5a5_479d_9c01_1bd588d88927.slice/crio-bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8 WatchSource:0}: Error finding container bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8: Status 404 returned error can't find the container with id bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8 Nov 01 22:27:15.738538 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.738515 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz"] Nov 01 22:27:15.836757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.836725 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.836907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.836768 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.836907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.836836 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6f9\" (UniqueName: \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.837498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.837473 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.877340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.877310 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.877759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.877736 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6f9\" (UniqueName: \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:15.992350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:15.992316 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:16.110224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.110178 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:16.188134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.188104 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" event={"ID":"b143f291-d5a5-479d-9c01-1bd588d88927","Type":"ContainerStarted","Data":"2826cde3fc985b746b858f1f0d3fdb9052626e4ba85a121167826c12ecef427c"} Nov 01 22:27:16.188304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.188164 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" event={"ID":"b143f291-d5a5-479d-9c01-1bd588d88927","Type":"ContainerStarted","Data":"bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8"} Nov 01 22:27:16.213110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.213058 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" podStartSLOduration=2.213044066 podStartE2EDuration="2.213044066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:16.206965058 +0000 UTC m=+2589.899945546" watchObservedRunningTime="2026-11-01 22:27:16.213044066 +0000 UTC m=+2589.906024551" Nov 01 22:27:16.337544 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:16.337508 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d31f95_36b9_4fb2_8d36_5fffaaec2d34.slice/crio-4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7 WatchSource:0}: Error finding container 4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7: Status 404 returned error can't find the container with id 4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7 Nov 01 22:27:16.340001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.339970 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl"] Nov 01 22:27:16.440940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.440911 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5c986c-7e0c-426e-80e2-fcce17350802" path="/var/lib/kubelet/pods/8b5c986c-7e0c-426e-80e2-fcce17350802/volumes" Nov 01 22:27:16.919069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:16.919040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:17.200598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.200563 5036 generic.go:356] "Generic (PLEG): container finished" podID="b143f291-d5a5-479d-9c01-1bd588d88927" containerID="2826cde3fc985b746b858f1f0d3fdb9052626e4ba85a121167826c12ecef427c" exitCode=0 Nov 01 22:27:17.201128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.200652 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" event={"ID":"b143f291-d5a5-479d-9c01-1bd588d88927","Type":"ContainerDied","Data":"2826cde3fc985b746b858f1f0d3fdb9052626e4ba85a121167826c12ecef427c"} Nov 01 22:27:17.202858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.202825 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" event={"ID":"85d31f95-36b9-4fb2-8d36-5fffaaec2d34","Type":"ContainerStarted","Data":"91dd0c41cf7470c768b6d7c26211bf96c0a9dad07f9dd122b5b0ca4c51a89975"} Nov 01 22:27:17.203020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.202882 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" event={"ID":"85d31f95-36b9-4fb2-8d36-5fffaaec2d34","Type":"ContainerStarted","Data":"4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7"} Nov 01 22:27:17.239354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.239297 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" podStartSLOduration=2.239281275 podStartE2EDuration="2.239281275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:17.23920726 +0000 UTC m=+2590.932187747" watchObservedRunningTime="2026-11-01 22:27:17.239281275 +0000 UTC m=+2590.932261764" Nov 01 22:27:17.707412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:17.705836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:18.212762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.212675 5036 generic.go:356] "Generic (PLEG): container finished" podID="85d31f95-36b9-4fb2-8d36-5fffaaec2d34" containerID="91dd0c41cf7470c768b6d7c26211bf96c0a9dad07f9dd122b5b0ca4c51a89975" exitCode=0 Nov 01 22:27:18.212762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.212712 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" event={"ID":"85d31f95-36b9-4fb2-8d36-5fffaaec2d34","Type":"ContainerDied","Data":"91dd0c41cf7470c768b6d7c26211bf96c0a9dad07f9dd122b5b0ca4c51a89975"} Nov 01 22:27:18.422206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.422164 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:18.549511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.538934 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\") pod \"b143f291-d5a5-479d-9c01-1bd588d88927\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " Nov 01 22:27:18.728351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.728307 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\" (UniqueName: \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\") pod \"b143f291-d5a5-479d-9c01-1bd588d88927\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " Nov 01 22:27:18.734800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.734757 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\") pod \"b143f291-d5a5-479d-9c01-1bd588d88927\" (UID: \"b143f291-d5a5-479d-9c01-1bd588d88927\") " Nov 01 22:27:19.014911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.623672 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca" pod "b143f291-d5a5-479d-9c01-1bd588d88927" (UID: "b143f291-d5a5-479d-9c01-1bd588d88927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:19.014911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:18.933354 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:19.017912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.017867 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:27:19.018074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.017942 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:27:19.019879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.019845 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca" pod "b143f291-d5a5-479d-9c01-1bd588d88927" (UID: "b143f291-d5a5-479d-9c01-1bd588d88927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:19.047718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.047676 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx" pod "b143f291-d5a5-479d-9c01-1bd588d88927" (UID: "b143f291-d5a5-479d-9c01-1bd588d88927"). InnerVolumeSpecName "kube-api-access-wqlsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:19.116532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.114230 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:19.116532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.114273 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b143f291-d5a5-479d-9c01-1bd588d88927-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:19.116532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.114290 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlsx\" (UniqueName: \"kubernetes.io/projected/b143f291-d5a5-479d-9c01-1bd588d88927-kube-api-access-wqlsx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:19.280723 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:19.280686 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:27:19.297119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.297069 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" Nov 01 22:27:19.315661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.315598 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz" event={"ID":"b143f291-d5a5-479d-9c01-1bd588d88927","Type":"ContainerDied","Data":"bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8"} Nov 01 22:27:19.315893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.315857 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc61b5f44ae4fc751bdbf62211c7c760f4d5b060783e13cdd755897bf68e02a8" Nov 01 22:27:19.566631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.550551 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:27:19.566631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.551289 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.566631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.551614 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.635468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.634730 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.635468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.635174 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.635753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.635494 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:27:19.639465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.636937 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.783431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.782617 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:19.819513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:19.812622 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:27:20.035478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.031552 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.371485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.356663 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.371485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.357820 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 22:27:20.423490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.420738 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.592467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.556694 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.592467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.557516 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 22:27:20.592467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.575396 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 22:27:20.644523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.637140 5036 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 01 22:27:20.644523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.637691 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.644523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.638015 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 22:27:20.644523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.639403 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 22:27:20.644523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.639701 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.657870 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.657928 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.657966 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.658000 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.658034 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.658070 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:27:20.731515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.658102 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:27:20.796391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.786336 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:20.796896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.796762 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:27:20.799470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.797733 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:20.809895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.809408 5036 generic.go:356] "Generic (PLEG): container finished" podID="59073435-6a6e-462a-a5d1-44aec4513092" containerID="df631f6682a0c274fc38cf510beb860a574cdab8164570b6c253b72872f78fd8" exitCode=0 Nov 01 22:27:20.809895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.809729 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerDied","Data":"df631f6682a0c274fc38cf510beb860a574cdab8164570b6c253b72872f78fd8"} Nov 01 22:27:20.809895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.809792 5036 scope.go:122] "RemoveContainer" containerID="c29ef76b87ed32778b7557552cbad1084021a6cd447e04d8e74e0b095f7b5a5c" Nov 01 22:27:20.833229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.824859 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 22:27:20.833229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.825244 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:27:20.932519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.932400 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:20.971868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.036088 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.036706 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.049179 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.058538 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:27:21.062204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.059301 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:27:21.063391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.063203 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:27:21.197089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.107970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\") pod \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " Nov 01 22:27:21.197089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.108099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\") pod \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " Nov 01 22:27:21.197089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.108184 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\" (UniqueName: \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\") pod \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\" (UID: \"85d31f95-36b9-4fb2-8d36-5fffaaec2d34\") " Nov 01 22:27:21.197089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.129067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca" pod "85d31f95-36b9-4fb2-8d36-5fffaaec2d34" (UID: "85d31f95-36b9-4fb2-8d36-5fffaaec2d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:21.251678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.212805 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.251678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.227394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:21.275966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.254395 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:27:21.281771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.281569 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca" pod "85d31f95-36b9-4fb2-8d36-5fffaaec2d34" (UID: "85d31f95-36b9-4fb2-8d36-5fffaaec2d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:21.281771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.281734 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9" pod "85d31f95-36b9-4fb2-8d36-5fffaaec2d34" (UID: "85d31f95-36b9-4fb2-8d36-5fffaaec2d34"). InnerVolumeSpecName "kube-api-access-ss6f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:21.297380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.296798 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:27:21.314964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.314297 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.314964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.314346 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6f9\" (UniqueName: \"kubernetes.io/projected/85d31f95-36b9-4fb2-8d36-5fffaaec2d34-kube-api-access-ss6f9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.375878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.375282 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:27:21.410464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.403567 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:21.460519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.456868 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:27:21.461023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.461005 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:27:21.481188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.475709 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:27:21.481188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.479248 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:27:21.485391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.485150 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:27:21.546147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.546112 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:27:21.567407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567366 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567437 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567554 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567634 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567730 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567825 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.567949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567931 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.568018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.567980 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.568091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.568033 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\") pod \"59073435-6a6e-462a-a5d1-44aec4513092\" (UID: \"59073435-6a6e-462a-a5d1-44aec4513092\") " Nov 01 22:27:21.569245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.569210 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:21.569878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.569848 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:21.570068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.570046 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:21.573270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.573244 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:21.573391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.573336 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:21.575061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.575025 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:21.575369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.575336 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:21.576705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.576680 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "kube-api-access-w7ppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:21.578387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.578353 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo" pod "59073435-6a6e-462a-a5d1-44aec4513092" (UID: "59073435-6a6e-462a-a5d1-44aec4513092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:27:21.670747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.670716 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.670983 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59073435-6a6e-462a-a5d1-44aec4513092-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671107 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671219 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671329 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ppj\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-kube-api-access-w7ppj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671435 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671580 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59073435-6a6e-462a-a5d1-44aec4513092-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671683 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59073435-6a6e-462a-a5d1-44aec4513092-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.671789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.671776 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59073435-6a6e-462a-a5d1-44aec4513092-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:21.832606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.831365 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" Nov 01 22:27:21.832606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.831609 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl" event={"ID":"85d31f95-36b9-4fb2-8d36-5fffaaec2d34","Type":"ContainerDied","Data":"4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7"} Nov 01 22:27:21.832606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.832207 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4376d5f08d43ed3e8bdf2dd9e1f44b3fda532979efe4b7c079228365a01273b7" Nov 01 22:27:21.835259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.835226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:21.837501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.837478 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" Nov 01 22:27:21.837635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.837528 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6" event={"ID":"59073435-6a6e-462a-a5d1-44aec4513092","Type":"ContainerDied","Data":"4138d58a8228d892d8b8615a14db9ca27e7bac25875d5202fd05d82c1bcc0ca2"} Nov 01 22:27:21.837635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.837580 5036 scope.go:122] "RemoveContainer" containerID="df631f6682a0c274fc38cf510beb860a574cdab8164570b6c253b72872f78fd8" Nov 01 22:27:21.972544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:21.922873 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:27:22.007429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:22.007386 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-829cb0-openshift-default-fd77c8645-xkkb6"] Nov 01 22:27:22.461785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:22.461742 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59073435-6a6e-462a-a5d1-44aec4513092" path="/var/lib/kubelet/pods/59073435-6a6e-462a-a5d1-44aec4513092/volumes" Nov 01 22:27:22.640512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:22.640473 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:22.826758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:22.826681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:22.962051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:22.961687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:23.121525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:23.121103 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:23.568468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:23.562801 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:24.028490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.028428 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k"] Nov 01 22:27:24.034267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.034234 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef8c9cd-mmp6k"] Nov 01 22:27:24.106469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.106343 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-4612e3"] Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107516 5036 cpu_manager.go:414] "containerMap: removing container" podUID="85d31f95-36b9-4fb2-8d36-5fffaaec2d34" containerName="revoke-keys" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107545 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d31f95-36b9-4fb2-8d36-5fffaaec2d34" containerName="revoke-keys" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107594 5036 cpu_manager.go:414] "containerMap: removing container" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107606 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107649 5036 cpu_manager.go:414] "containerMap: removing container" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107659 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107668 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b143f291-d5a5-479d-9c01-1bd588d88927" containerName="revoke-keys" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107676 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b143f291-d5a5-479d-9c01-1bd588d88927" containerName="revoke-keys" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107850 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d31f95-36b9-4fb2-8d36-5fffaaec2d34" containerName="revoke-keys" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107872 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:24.110510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.107888 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b143f291-d5a5-479d-9c01-1bd588d88927" containerName="revoke-keys" Nov 01 22:27:24.134754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.134723 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-4612e3"] Nov 01 22:27:24.134950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.134832 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:24.146888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.146862 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 22:27:24.266805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.266771 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtfg\" (UniqueName: \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\") pod \"test-curl-6953-4612e3\" (UID: \"70bb6396-2e57-4303-a9c4-30112b574904\") " pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:24.368319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.368200 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtfg\" (UniqueName: \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\") pod \"test-curl-6953-4612e3\" (UID: \"70bb6396-2e57-4303-a9c4-30112b574904\") " pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:24.413749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.413689 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtfg\" (UniqueName: \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\") pod \"test-curl-6953-4612e3\" (UID: \"70bb6396-2e57-4303-a9c4-30112b574904\") " pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:24.442863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.442824 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40daf00-247f-48e8-82b5-8f73e6c4a8d3" path="/var/lib/kubelet/pods/c40daf00-247f-48e8-82b5-8f73e6c4a8d3/volumes" Nov 01 22:27:24.457609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.457583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:24.457846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.457609 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:24.898465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:24.892996 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-4612e3"] Nov 01 22:27:25.036481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.033776 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:25.043576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.043546 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz"] Nov 01 22:27:25.050331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.050307 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c126bc0-h4szz"] Nov 01 22:27:25.272470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.270007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:25.721179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.721073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:25.907587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.907553 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:25.966620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.965810 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-4612e3" event={"ID":"70bb6396-2e57-4303-a9c4-30112b574904","Type":"ContainerStarted","Data":"765f6c16ae24bd00716d40ddd2a9ab4fa56579cffe8055085e38794d6c687b23"} Nov 01 22:27:25.966620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.965875 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-4612e3" event={"ID":"70bb6396-2e57-4303-a9c4-30112b574904","Type":"ContainerStarted","Data":"4b7623ccc7a312622a95cfa06e445db54fd20bdefeb6a1e37729cc2d1f79c18f"} Nov 01 22:27:25.994522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:25.991284 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-4612e3" podStartSLOduration=1.9912673189999999 podStartE2EDuration="1.991267319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:25.990790545 +0000 UTC m=+2599.683771033" watchObservedRunningTime="2026-11-01 22:27:25.991267319 +0000 UTC m=+2599.684247806" Nov 01 22:27:26.043876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:26.043843 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:26.448506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:26.448466 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cd34dc-7c46-4a3b-b953-2cfe5f1bba19" path="/var/lib/kubelet/pods/19cd34dc-7c46-4a3b-b953-2cfe5f1bba19/volumes" Nov 01 22:27:26.845464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:26.839108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:27.177533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.176499 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-4612e3"] Nov 01 22:27:27.203673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.203638 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-4612e3"] Nov 01 22:27:27.322475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.322425 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6953-ddf9f6"] Nov 01 22:27:27.323348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.323314 5036 cpu_manager.go:414] "containerMap: removing container" podUID="70bb6396-2e57-4303-a9c4-30112b574904" containerName="test-curl-6953-4612e3" Nov 01 22:27:27.323348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.323342 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bb6396-2e57-4303-a9c4-30112b574904" containerName="test-curl-6953-4612e3" Nov 01 22:27:27.323570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.323497 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bb6396-2e57-4303-a9c4-30112b574904" containerName="test-curl-6953-4612e3" Nov 01 22:27:27.323570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.323510 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="59073435-6a6e-462a-a5d1-44aec4513092" containerName="istio-proxy" Nov 01 22:27:27.332950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.332925 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-ddf9f6"] Nov 01 22:27:27.333099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.333021 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:27.524036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.523997 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rpv\" (UniqueName: \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\") pod \"test-curl-6953-ddf9f6\" (UID: \"53cf7e51-6477-4b03-9024-c3d90540d4ae\") " pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:27.536900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.536873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:27.627470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.626693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rpv\" (UniqueName: \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\") pod \"test-curl-6953-ddf9f6\" (UID: \"53cf7e51-6477-4b03-9024-c3d90540d4ae\") " pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:27.651114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.651070 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rpv\" (UniqueName: \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\") pod \"test-curl-6953-ddf9f6\" (UID: \"53cf7e51-6477-4b03-9024-c3d90540d4ae\") " pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:27.651326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:27.651153 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:28.018435 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:28.018312 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53cf7e51_6477_4b03_9024_c3d90540d4ae.slice/crio-167b5eb1169837a8a5203ef6c42e4be893580f176a82fe12084c9ab1420fa7ad WatchSource:0}: Error finding container 167b5eb1169837a8a5203ef6c42e4be893580f176a82fe12084c9ab1420fa7ad: Status 404 returned error can't find the container with id 167b5eb1169837a8a5203ef6c42e4be893580f176a82fe12084c9ab1420fa7ad Nov 01 22:27:28.018435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.018379 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-4612e3" podUID="70bb6396-2e57-4303-a9c4-30112b574904" containerName="test-curl-6953-4612e3" containerID="cri-o://765f6c16ae24bd00716d40ddd2a9ab4fa56579cffe8055085e38794d6c687b23" gracePeriod=2 Nov 01 22:27:28.020201 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:28.020167 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-4612e3\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="70bb6396-2e57-4303-a9c4-30112b574904" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:28.022663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.022639 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6953-ddf9f6"] Nov 01 22:27:28.245942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.245909 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:28.648354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.648168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:28.778927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.778892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:28.950520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:28.947791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:29.126468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.122164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:29.143467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.140756 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-ddf9f6" event={"ID":"53cf7e51-6477-4b03-9024-c3d90540d4ae","Type":"ContainerStarted","Data":"e75339408ed19c98c24824b8611d98a65000e0ea4714d90d8f59adda3dcb317e"} Nov 01 22:27:29.143467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.140815 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6953-ddf9f6" event={"ID":"53cf7e51-6477-4b03-9024-c3d90540d4ae","Type":"ContainerStarted","Data":"167b5eb1169837a8a5203ef6c42e4be893580f176a82fe12084c9ab1420fa7ad"} Nov 01 22:27:29.591790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.591681 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6953-ddf9f6" podStartSLOduration=2.591650611 podStartE2EDuration="2.591650611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:29.179993386 +0000 UTC m=+2602.872973874" watchObservedRunningTime="2026-11-01 22:27:29.591650611 +0000 UTC m=+2603.284631098" Nov 01 22:27:29.605992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.605948 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:27:29.606468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.606413 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" podUID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" containerName="maas-api" containerID="cri-o://56c18eefb5eea0e82ab86d82bbff3f6b13355fa2d2792ff3b30eafa872f8dd6e" gracePeriod=30 Nov 01 22:27:29.720467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.709758 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:29.908521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:29.905812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:30.258530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.253930 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl"] Nov 01 22:27:30.290818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.290583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:30.297274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.297180 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d6e25f-5kkpl"] Nov 01 22:27:30.306321 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.306239 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6953-ddf9f6"] Nov 01 22:27:30.309721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.309414 5036 generic.go:356] "Generic (PLEG): container finished" podID="70bb6396-2e57-4303-a9c4-30112b574904" containerID="765f6c16ae24bd00716d40ddd2a9ab4fa56579cffe8055085e38794d6c687b23" exitCode=137 Nov 01 22:27:30.312199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.311815 5036 generic.go:356] "Generic (PLEG): container finished" podID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" containerID="56c18eefb5eea0e82ab86d82bbff3f6b13355fa2d2792ff3b30eafa872f8dd6e" exitCode=0 Nov 01 22:27:30.312979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.312886 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" event={"ID":"3960c33e-59ee-461b-91ce-9cd2a6ab32ac","Type":"ContainerDied","Data":"56c18eefb5eea0e82ab86d82bbff3f6b13355fa2d2792ff3b30eafa872f8dd6e"} Nov 01 22:27:30.353252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.353191 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6953-ddf9f6"] Nov 01 22:27:30.422721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.422072 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:30.454131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.453585 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d31f95-36b9-4fb2-8d36-5fffaaec2d34" path="/var/lib/kubelet/pods/85d31f95-36b9-4fb2-8d36-5fffaaec2d34/volumes" Nov 01 22:27:30.525762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.525735 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:27:30.532091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.532061 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg"] Nov 01 22:27:30.533186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533157 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" containerName="maas-api" Nov 01 22:27:30.533186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533182 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" containerName="maas-api" Nov 01 22:27:30.533345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533215 5036 cpu_manager.go:414] "containerMap: removing container" podUID="53cf7e51-6477-4b03-9024-c3d90540d4ae" containerName="test-curl-6953-ddf9f6" Nov 01 22:27:30.533345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533224 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cf7e51-6477-4b03-9024-c3d90540d4ae" containerName="test-curl-6953-ddf9f6" Nov 01 22:27:30.533438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533425 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" containerName="maas-api" Nov 01 22:27:30.533521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.533465 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cf7e51-6477-4b03-9024-c3d90540d4ae" containerName="test-curl-6953-ddf9f6" Nov 01 22:27:30.538255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.538228 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.543364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.543339 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:27:30.543663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.543639 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:27:30.544689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.544530 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg"] Nov 01 22:27:30.552930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.552907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:30.565306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.565228 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:30.695814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.695781 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\") pod \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " Nov 01 22:27:30.696296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.695868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\" (UniqueName: \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\") pod \"70bb6396-2e57-4303-a9c4-30112b574904\" (UID: \"70bb6396-2e57-4303-a9c4-30112b574904\") " Nov 01 22:27:30.696296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.695955 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") pod \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " Nov 01 22:27:30.696296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.696059 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\") pod \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " Nov 01 22:27:30.696296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.696153 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\" (UniqueName: \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\") pod \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\" (UID: \"3960c33e-59ee-461b-91ce-9cd2a6ab32ac\") " Nov 01 22:27:30.696632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.696600 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.697054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.696906 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsk8x\" (UniqueName: \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.697054 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.696966 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.697666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.697642 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca" pod "3960c33e-59ee-461b-91ce-9cd2a6ab32ac" (UID: "3960c33e-59ee-461b-91ce-9cd2a6ab32ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:30.698567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.698529 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls" pod "3960c33e-59ee-461b-91ce-9cd2a6ab32ac" (UID: "3960c33e-59ee-461b-91ce-9cd2a6ab32ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:30.698948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.698919 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs" pod "3960c33e-59ee-461b-91ce-9cd2a6ab32ac" (UID: "3960c33e-59ee-461b-91ce-9cd2a6ab32ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:30.699066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.699039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg" pod "70bb6396-2e57-4303-a9c4-30112b574904" (UID: "70bb6396-2e57-4303-a9c4-30112b574904"). InnerVolumeSpecName "kube-api-access-kmtfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:30.700143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.700119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv" pod "3960c33e-59ee-461b-91ce-9cd2a6ab32ac" (UID: "3960c33e-59ee-461b-91ce-9cd2a6ab32ac"). InnerVolumeSpecName "kube-api-access-dbvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:30.798422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798382 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsk8x\" (UniqueName: \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.798648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798469 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.798648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798620 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.798802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798715 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:30.798802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798735 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtfg\" (UniqueName: \"kubernetes.io/projected/70bb6396-2e57-4303-a9c4-30112b574904-kube-api-access-kmtfg\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:30.798802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798749 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:30.798802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798763 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:30.798802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.798780 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvkv\" (UniqueName: \"kubernetes.io/projected/3960c33e-59ee-461b-91ce-9cd2a6ab32ac-kube-api-access-dbvkv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:30.799415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.799343 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.801591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.801563 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.810894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.810869 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsk8x\" (UniqueName: \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\") pod \"maas-api-revoke-sub-a5b9d752-blwhg\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:30.873373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.873341 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:30.878238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:30.878211 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:31.234707 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:31.234657 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be22d27_675b_4284_89cb_c522b1c2501e.slice/crio-8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06 WatchSource:0}: Error finding container 8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06: Status 404 returned error can't find the container with id 8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06 Nov 01 22:27:31.235287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.235148 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg"] Nov 01 22:27:31.324532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.324487 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" event={"ID":"8be22d27-675b-4284-89cb-c522b1c2501e","Type":"ContainerStarted","Data":"8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06"} Nov 01 22:27:31.326417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.326392 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:31.326603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.326438 5036 scope.go:122] "RemoveContainer" containerID="765f6c16ae24bd00716d40ddd2a9ab4fa56579cffe8055085e38794d6c687b23" Nov 01 22:27:31.330361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.330333 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" Nov 01 22:27:31.330489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.330357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5" event={"ID":"3960c33e-59ee-461b-91ce-9cd2a6ab32ac","Type":"ContainerDied","Data":"49f05d86d7ed769c2e17de374f0a038f2c2547f5d6146acc460dd2c6ccbba34c"} Nov 01 22:27:31.331276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.330764 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6953-ddf9f6" podUID="53cf7e51-6477-4b03-9024-c3d90540d4ae" containerName="test-curl-6953-ddf9f6" containerID="cri-o://e75339408ed19c98c24824b8611d98a65000e0ea4714d90d8f59adda3dcb317e" gracePeriod=2 Nov 01 22:27:31.342994 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:31.342972 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-4612e3\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="70bb6396-2e57-4303-a9c4-30112b574904" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:31.343612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.343595 5036 scope.go:122] "RemoveContainer" containerID="56c18eefb5eea0e82ab86d82bbff3f6b13355fa2d2792ff3b30eafa872f8dd6e" Nov 01 22:27:31.361146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.361121 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:27:31.361601 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:31.361581 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-4612e3\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="70bb6396-2e57-4303-a9c4-30112b574904" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:31.365755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.365734 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d6e25f-5cb477bc88-h6ts5"] Nov 01 22:27:31.547337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:31.547303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:32.199130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.199044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:32.347518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.347475 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" event={"ID":"8be22d27-675b-4284-89cb-c522b1c2501e","Type":"ContainerStarted","Data":"4f284c4f2fc86fdc66c9e1a03a68c4fe270c1d77470265a3100695b1812021a3"} Nov 01 22:27:32.350605 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:32.350563 5036 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6953-4612e3\" is forbidden: User \"system:node:ip-10-0-131-110.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.us-east-2.compute.internal' and this object" podUID="70bb6396-2e57-4303-a9c4-30112b574904" pod="opendatahub/test-curl-6953-4612e3" Nov 01 22:27:32.368856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.368810 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" podStartSLOduration=2.368794958 podStartE2EDuration="2.368794958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:32.368303234 +0000 UTC m=+2606.061283718" watchObservedRunningTime="2026-11-01 22:27:32.368794958 +0000 UTC m=+2606.061775444" Nov 01 22:27:32.439814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.439781 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3960c33e-59ee-461b-91ce-9cd2a6ab32ac" path="/var/lib/kubelet/pods/3960c33e-59ee-461b-91ce-9cd2a6ab32ac/volumes" Nov 01 22:27:32.440331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.440298 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bb6396-2e57-4303-a9c4-30112b574904" path="/var/lib/kubelet/pods/70bb6396-2e57-4303-a9c4-30112b574904/volumes" Nov 01 22:27:32.905669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:32.905630 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:33.362940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.362904 5036 generic.go:356] "Generic (PLEG): container finished" podID="53cf7e51-6477-4b03-9024-c3d90540d4ae" containerID="e75339408ed19c98c24824b8611d98a65000e0ea4714d90d8f59adda3dcb317e" exitCode=137 Nov 01 22:27:33.366488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.366420 5036 generic.go:356] "Generic (PLEG): container finished" podID="8be22d27-675b-4284-89cb-c522b1c2501e" containerID="4f284c4f2fc86fdc66c9e1a03a68c4fe270c1d77470265a3100695b1812021a3" exitCode=0 Nov 01 22:27:33.366628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.366595 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" event={"ID":"8be22d27-675b-4284-89cb-c522b1c2501e","Type":"ContainerDied","Data":"4f284c4f2fc86fdc66c9e1a03a68c4fe270c1d77470265a3100695b1812021a3"} Nov 01 22:27:33.579547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.579521 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:33.710370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.710272 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:33.762465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.761645 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\" (UniqueName: \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\") pod \"53cf7e51-6477-4b03-9024-c3d90540d4ae\" (UID: \"53cf7e51-6477-4b03-9024-c3d90540d4ae\") " Nov 01 22:27:33.776122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.775712 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv" pod "53cf7e51-6477-4b03-9024-c3d90540d4ae" (UID: "53cf7e51-6477-4b03-9024-c3d90540d4ae"). InnerVolumeSpecName "kube-api-access-w9rpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:33.863736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:33.863707 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rpv\" (UniqueName: \"kubernetes.io/projected/53cf7e51-6477-4b03-9024-c3d90540d4ae-kube-api-access-w9rpv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:34.376826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.376791 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6953-ddf9f6" Nov 01 22:27:34.376826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.376817 5036 scope.go:122] "RemoveContainer" containerID="e75339408ed19c98c24824b8611d98a65000e0ea4714d90d8f59adda3dcb317e" Nov 01 22:27:34.379117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.379089 5036 generic.go:356] "Generic (PLEG): container finished" podID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerID="629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8" exitCode=0 Nov 01 22:27:34.379271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.379116 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerDied","Data":"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8"} Nov 01 22:27:34.379714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.379694 5036 scope.go:122] "RemoveContainer" containerID="629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8" Nov 01 22:27:34.381564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.381514 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:34.381682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.381617 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:34.381682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.381668 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:34.456785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.441791 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cf7e51-6477-4b03-9024-c3d90540d4ae" path="/var/lib/kubelet/pods/53cf7e51-6477-4b03-9024-c3d90540d4ae/volumes" Nov 01 22:27:34.561328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.558513 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:34.704511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.702128 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:34.830747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.830708 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\") pod \"8be22d27-675b-4284-89cb-c522b1c2501e\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " Nov 01 22:27:34.830958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.830770 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\") pod \"8be22d27-675b-4284-89cb-c522b1c2501e\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " Nov 01 22:27:34.830958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.830893 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\" (UniqueName: \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\") pod \"8be22d27-675b-4284-89cb-c522b1c2501e\" (UID: \"8be22d27-675b-4284-89cb-c522b1c2501e\") " Nov 01 22:27:34.831225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.831195 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca" pod "8be22d27-675b-4284-89cb-c522b1c2501e" (UID: "8be22d27-675b-4284-89cb-c522b1c2501e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:34.831634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.831612 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:34.832990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.832969 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca" pod "8be22d27-675b-4284-89cb-c522b1c2501e" (UID: "8be22d27-675b-4284-89cb-c522b1c2501e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:34.832990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.832979 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x" pod "8be22d27-675b-4284-89cb-c522b1c2501e" (UID: "8be22d27-675b-4284-89cb-c522b1c2501e"). InnerVolumeSpecName "kube-api-access-xsk8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:34.932928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.932835 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsk8x\" (UniqueName: \"kubernetes.io/projected/8be22d27-675b-4284-89cb-c522b1c2501e-kube-api-access-xsk8x\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:34.932928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:34.932865 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be22d27-675b-4284-89cb-c522b1c2501e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:35.115681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.115642 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:35.271596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.270103 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz"] Nov 01 22:27:35.271596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.270163 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-702002-k5clz"] Nov 01 22:27:35.271596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.271490 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:35.447471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.444661 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:35.757522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.754810 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerStarted","Data":"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc"} Nov 01 22:27:35.757522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.757499 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:35.791468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.760415 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" event={"ID":"8be22d27-675b-4284-89cb-c522b1c2501e","Type":"ContainerDied","Data":"8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06"} Nov 01 22:27:35.791468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.760485 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb6c8f9d845e60d7506f78ad70f304518735cc3ad173fd4e79fd2f0ae704b06" Nov 01 22:27:35.791468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:35.760555 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg" Nov 01 22:27:36.151880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.151849 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:27:36.153168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.153134 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:36.153337 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.153192 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:36.224227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.224181 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:36.320929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.320892 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:27:36.440665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.440589 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b143f291-d5a5-479d-9c01-1bd588d88927" path="/var/lib/kubelet/pods/b143f291-d5a5-479d-9c01-1bd588d88927/volumes" Nov 01 22:27:36.869254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:36.869216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:37.011473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.002543 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:37.152794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.151934 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:37.152794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.151992 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:37.167188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.167003 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:27:37.167358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.167239 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" containerID="cri-o://bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976" gracePeriod=30 Nov 01 22:27:37.239083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.234169 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:37.721818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.721778 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc"] Nov 01 22:27:37.722528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.722508 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8be22d27-675b-4284-89cb-c522b1c2501e" containerName="revoke-keys" Nov 01 22:27:37.722528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.722526 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be22d27-675b-4284-89cb-c522b1c2501e" containerName="revoke-keys" Nov 01 22:27:37.722701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.722671 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be22d27-675b-4284-89cb-c522b1c2501e" containerName="revoke-keys" Nov 01 22:27:37.727235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.727211 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.735150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.735127 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc"] Nov 01 22:27:37.739818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.739791 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:27:37.739954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.739855 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:27:37.750124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.750088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.750246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.750184 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2xj\" (UniqueName: \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.750326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.750297 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.836324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.836294 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:37.863474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.858647 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.863474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.858825 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2xj\" (UniqueName: \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.863474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.858929 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.863474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.860287 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.921547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.921507 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2xj\" (UniqueName: \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:37.922131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:37.921828 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:38.046245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.046211 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:38.161460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.154828 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:38.161460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.154920 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:38.375914 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:38.375879 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7522fd75_37df_410a_b235_ea3f3a91c48b.slice/crio-33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd WatchSource:0}: Error finding container 33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd: Status 404 returned error can't find the container with id 33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd Nov 01 22:27:38.405665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.377087 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc"] Nov 01 22:27:38.712528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.708010 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:38.829826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:38.829790 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" event={"ID":"7522fd75-37df-410a-b235-ea3f3a91c48b","Type":"ContainerStarted","Data":"33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd"} Nov 01 22:27:39.126808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.126761 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:39.155507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.152829 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:39.155507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.152911 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:39.268481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.267305 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:39.904524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.901529 5036 generic.go:356] "Generic (PLEG): container finished" podID="7522fd75-37df-410a-b235-ea3f3a91c48b" containerID="6ac63908d91a97813c19d5197365a1579c9a19264d321bc5960869e7275e76be" exitCode=0 Nov 01 22:27:39.904524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.901800 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" event={"ID":"7522fd75-37df-410a-b235-ea3f3a91c48b","Type":"ContainerDied","Data":"6ac63908d91a97813c19d5197365a1579c9a19264d321bc5960869e7275e76be"} Nov 01 22:27:39.918506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:39.912866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:40.152182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.152140 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:40.152624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.152205 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:40.641140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.641110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:40.831739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.831702 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx"] Nov 01 22:27:40.836553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.836533 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:40.844845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.844818 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx"] Nov 01 22:27:40.925844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.925804 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqfz\" (UniqueName: \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:40.926007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.925885 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:40.926189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:40.926161 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.029025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.028989 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.029456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.029411 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqfz\" (UniqueName: \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.029668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.029648 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.032839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.032789 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.041770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.041723 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.046999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.046949 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqfz\" (UniqueName: \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\") pod \"maas-api-revoke-sub-3ba37df5-6chqx\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.151930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.151897 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Nov 01 22:27:41.152047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.151959 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Nov 01 22:27:41.155218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.155198 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:41.166243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.166221 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:41.168644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.168612 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:27:41.168739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.168673 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:27:41.199823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.199783 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2"] Nov 01 22:27:41.201241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.201213 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7522fd75-37df-410a-b235-ea3f3a91c48b" containerName="revoke-keys" Nov 01 22:27:41.201821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.201799 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7522fd75-37df-410a-b235-ea3f3a91c48b" containerName="revoke-keys" Nov 01 22:27:41.204803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.204618 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7522fd75-37df-410a-b235-ea3f3a91c48b" containerName="revoke-keys" Nov 01 22:27:41.219246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.211603 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.232825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.232468 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\") pod \"7522fd75-37df-410a-b235-ea3f3a91c48b\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " Nov 01 22:27:41.232825 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.232558 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\") pod \"7522fd75-37df-410a-b235-ea3f3a91c48b\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " Nov 01 22:27:41.234179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.233103 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\" (UniqueName: \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\") pod \"7522fd75-37df-410a-b235-ea3f3a91c48b\" (UID: \"7522fd75-37df-410a-b235-ea3f3a91c48b\") " Nov 01 22:27:41.237717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.237690 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca" pod "7522fd75-37df-410a-b235-ea3f3a91c48b" (UID: "7522fd75-37df-410a-b235-ea3f3a91c48b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:41.239229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.239154 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2"] Nov 01 22:27:41.239588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.239545 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj" pod "7522fd75-37df-410a-b235-ea3f3a91c48b" (UID: "7522fd75-37df-410a-b235-ea3f3a91c48b"). InnerVolumeSpecName "kube-api-access-kf2xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:41.242232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.242208 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca" pod "7522fd75-37df-410a-b235-ea3f3a91c48b" (UID: "7522fd75-37df-410a-b235-ea3f3a91c48b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:41.335729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.335687 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.335940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.335845 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.336018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.335959 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf99f\" (UniqueName: \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.336118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.336097 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2xj\" (UniqueName: \"kubernetes.io/projected/7522fd75-37df-410a-b235-ea3f3a91c48b-kube-api-access-kf2xj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:41.336197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.336125 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:41.336197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.336142 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7522fd75-37df-410a-b235-ea3f3a91c48b-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:41.437760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.437674 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf99f\" (UniqueName: \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.437924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.437847 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.437986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.437967 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.438612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.438583 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.440353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.440333 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.449985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.449960 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf99f\" (UniqueName: \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\") pod \"maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.503991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.503531 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx"] Nov 01 22:27:41.514147 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:41.514105 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f688b46_aa57_47f7_b25a_c4934dbc1c92.slice/crio-883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92 WatchSource:0}: Error finding container 883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92: Status 404 returned error can't find the container with id 883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92 Nov 01 22:27:41.546788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.546760 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:41.742985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.742941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:41.932277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.932238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" event={"ID":"1f688b46-aa57-47f7-b25a-c4934dbc1c92","Type":"ContainerStarted","Data":"883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92"} Nov 01 22:27:41.934020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.933991 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" Nov 01 22:27:41.934020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.934005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc" event={"ID":"7522fd75-37df-410a-b235-ea3f3a91c48b","Type":"ContainerDied","Data":"33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd"} Nov 01 22:27:41.934201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.934050 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bda6c397dd3e07cd4049fe902f5f4257d40dc62e9c27f013b89b6b8d8768bd" Nov 01 22:27:41.936583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:41.936555 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2"] Nov 01 22:27:41.979567 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:41.979538 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ca22f7_106e_49f0_bbaf_b11b29145d79.slice/crio-be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a WatchSource:0}: Error finding container be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a: Status 404 returned error can't find the container with id be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a Nov 01 22:27:42.157255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.157225 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:27:42.158378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.158356 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:27:42.623952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.623913 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:42.773946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.773859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:42.875552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.874432 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:27:42.875552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.874752 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" containerID="cri-o://06a46c50e7c6bbc5a12f32312687ea50fe4c1e35373bc22debd55268f618fbc2" gracePeriod=30 Nov 01 22:27:42.927628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.927593 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:27:42.932762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.932727 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:42.948385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.948358 5036 generic.go:356] "Generic (PLEG): container finished" podID="1f688b46-aa57-47f7-b25a-c4934dbc1c92" containerID="d1260bb4cabecfbb742f9084a829f943dd4fcf0d5f1952c17002b4e6276f2e78" exitCode=0 Nov 01 22:27:42.948637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.948507 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" event={"ID":"1f688b46-aa57-47f7-b25a-c4934dbc1c92","Type":"ContainerDied","Data":"d1260bb4cabecfbb742f9084a829f943dd4fcf0d5f1952c17002b4e6276f2e78"} Nov 01 22:27:42.982802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.951083 5036 generic.go:356] "Generic (PLEG): container finished" podID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerID="bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976" exitCode=0 Nov 01 22:27:42.982802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.951352 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" event={"ID":"9641db1e-73b2-482b-b72e-c05903aaed8d","Type":"ContainerDied","Data":"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976"} Nov 01 22:27:42.982802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.951404 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" event={"ID":"9641db1e-73b2-482b-b72e-c05903aaed8d","Type":"ContainerDied","Data":"7d7dfa576b5c3682fb527a2790fee13ef7611c90cc6715e2e14486cf1fe98331"} Nov 01 22:27:42.982802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.957816 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4" Nov 01 22:27:43.076475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:42.951436 5036 scope.go:122] "RemoveContainer" containerID="bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976" Nov 01 22:27:43.108950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.091094 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" event={"ID":"58ca22f7-106e-49f0-bbaf-b11b29145d79","Type":"ContainerStarted","Data":"61dd6ad8d2bda6534f814d0bdbd81ea11f485091fd927c5677c86cd0581681fd"} Nov 01 22:27:43.108950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.091149 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" event={"ID":"58ca22f7-106e-49f0-bbaf-b11b29145d79","Type":"ContainerStarted","Data":"be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a"} Nov 01 22:27:43.108950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.108800 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.108950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.108899 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.108987 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109028 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109104 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109243 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.109409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109325 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.114492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.109479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\") pod \"9641db1e-73b2-482b-b72e-c05903aaed8d\" (UID: \"9641db1e-73b2-482b-b72e-c05903aaed8d\") " Nov 01 22:27:43.149512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.147327 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:43.149512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.147539 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:43.149512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.148489 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:43.149512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.148659 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:43.149512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.148765 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:27:43.182470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.158062 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" podStartSLOduration=2.158043945 podStartE2EDuration="2.158043945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:43.149179879 +0000 UTC m=+2616.842160366" watchObservedRunningTime="2026-11-01 22:27:43.158043945 +0000 UTC m=+2616.851024431" Nov 01 22:27:43.182470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.158415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220688 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220723 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220740 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220754 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220769 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.227466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.220786 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.273520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.268409 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:43.288869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.283714 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:43.291751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.290898 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr" pod "9641db1e-73b2-482b-b72e-c05903aaed8d" (UID: "9641db1e-73b2-482b-b72e-c05903aaed8d"). InnerVolumeSpecName "kube-api-access-dbssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:43.310381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.310079 5036 scope.go:122] "RemoveContainer" containerID="bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976" Nov 01 22:27:43.311008 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:43.310980 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976\": container with ID starting with bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976 not found: ID does not exist" containerID="bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976" Nov 01 22:27:43.311185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.311013 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976"} err="failed to get container status \"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976\": rpc error: code = NotFound desc = could not find container \"bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976\": container with ID starting with bba3707a4051cfd388b742cf1b6e95222d966712ac23750aea91a943ac8ae976 not found: ID does not exist" Nov 01 22:27:43.322869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.322844 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbssr\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-kube-api-access-dbssr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.322999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.322871 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641db1e-73b2-482b-b72e-c05903aaed8d-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.322999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.322883 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641db1e-73b2-482b-b72e-c05903aaed8d-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:43.328193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.328168 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9"] Nov 01 22:27:43.328915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.328898 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" Nov 01 22:27:43.328915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.328915 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" Nov 01 22:27:43.329066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.329012 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" containerName="istio-proxy" Nov 01 22:27:43.334434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.334411 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.342701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.342677 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9"] Nov 01 22:27:43.424645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.424570 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96x7k\" (UniqueName: \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.424789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.424710 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.424964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.424938 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.520755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.520711 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:43.526222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.526192 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.526328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.526272 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96x7k\" (UniqueName: \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.526393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.526344 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.527046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.527020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.528753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.528723 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.565473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.564392 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96x7k\" (UniqueName: \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.619876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.619829 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:27:43.624830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.624801 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-vzdq4"] Nov 01 22:27:43.655373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.655348 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:43.694845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.694772 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:27:43.694845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.694829 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:27:43.945023 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:43.944907 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a2881b_6f4c_4b4a_98ac_86399cddff02.slice/crio-19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c WatchSource:0}: Error finding container 19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c: Status 404 returned error can't find the container with id 19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c Nov 01 22:27:43.945522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:43.945493 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9"] Nov 01 22:27:44.101711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.101676 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" event={"ID":"19a2881b-6f4c-4b4a-98ac-86399cddff02","Type":"ContainerStarted","Data":"19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c"} Nov 01 22:27:44.103274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.103248 5036 generic.go:356] "Generic (PLEG): container finished" podID="58ca22f7-106e-49f0-bbaf-b11b29145d79" containerID="61dd6ad8d2bda6534f814d0bdbd81ea11f485091fd927c5677c86cd0581681fd" exitCode=0 Nov 01 22:27:44.103400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.103331 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" event={"ID":"58ca22f7-106e-49f0-bbaf-b11b29145d79","Type":"ContainerDied","Data":"61dd6ad8d2bda6534f814d0bdbd81ea11f485091fd927c5677c86cd0581681fd"} Nov 01 22:27:44.367497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.367464 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:44.429274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.429244 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:44.454483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.441161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\") pod \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " Nov 01 22:27:44.454483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.441975 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\" (UniqueName: \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\") pod \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " Nov 01 22:27:44.454483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.442177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\") pod \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\" (UID: \"1f688b46-aa57-47f7-b25a-c4934dbc1c92\") " Nov 01 22:27:44.454483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.443119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca" pod "1f688b46-aa57-47f7-b25a-c4934dbc1c92" (UID: "1f688b46-aa57-47f7-b25a-c4934dbc1c92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:44.470891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.468732 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:44.489475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.477071 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9641db1e-73b2-482b-b72e-c05903aaed8d" path="/var/lib/kubelet/pods/9641db1e-73b2-482b-b72e-c05903aaed8d/volumes" Nov 01 22:27:44.523711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.523632 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz" pod "1f688b46-aa57-47f7-b25a-c4934dbc1c92" (UID: "1f688b46-aa57-47f7-b25a-c4934dbc1c92"). InnerVolumeSpecName "kube-api-access-znqfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:44.523711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.523631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca" pod "1f688b46-aa57-47f7-b25a-c4934dbc1c92" (UID: "1f688b46-aa57-47f7-b25a-c4934dbc1c92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:44.570127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.570087 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f688b46-aa57-47f7-b25a-c4934dbc1c92-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:44.570127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:44.570127 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znqfz\" (UniqueName: \"kubernetes.io/projected/1f688b46-aa57-47f7-b25a-c4934dbc1c92-kube-api-access-znqfz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:45.113727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.113701 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" Nov 01 22:27:45.113922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.113733 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx" event={"ID":"1f688b46-aa57-47f7-b25a-c4934dbc1c92","Type":"ContainerDied","Data":"883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92"} Nov 01 22:27:45.113922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.113780 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883cad5ce0f9c6081a0c6a552214953456e22b864123aeb43c608ce091681b92" Nov 01 22:27:45.115692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.115664 5036 generic.go:356] "Generic (PLEG): container finished" podID="19a2881b-6f4c-4b4a-98ac-86399cddff02" containerID="1fdc8209c919232580149c6ffe35c4277df3505ef2b2376c6c17e84a008a2a64" exitCode=0 Nov 01 22:27:45.115823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.115741 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" event={"ID":"19a2881b-6f4c-4b4a-98ac-86399cddff02","Type":"ContainerDied","Data":"1fdc8209c919232580149c6ffe35c4277df3505ef2b2376c6c17e84a008a2a64"} Nov 01 22:27:45.312346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.312318 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:45.366331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.366287 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:45.388073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.385478 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\" (UniqueName: \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\") pod \"58ca22f7-106e-49f0-bbaf-b11b29145d79\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " Nov 01 22:27:45.388073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.385632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\") pod \"58ca22f7-106e-49f0-bbaf-b11b29145d79\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " Nov 01 22:27:45.388073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.385927 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\") pod \"58ca22f7-106e-49f0-bbaf-b11b29145d79\" (UID: \"58ca22f7-106e-49f0-bbaf-b11b29145d79\") " Nov 01 22:27:45.393171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.393137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca" pod "58ca22f7-106e-49f0-bbaf-b11b29145d79" (UID: "58ca22f7-106e-49f0-bbaf-b11b29145d79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:45.439555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.439438 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca" pod "58ca22f7-106e-49f0-bbaf-b11b29145d79" (UID: "58ca22f7-106e-49f0-bbaf-b11b29145d79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:45.465648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.465576 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f" pod "58ca22f7-106e-49f0-bbaf-b11b29145d79" (UID: "58ca22f7-106e-49f0-bbaf-b11b29145d79"). InnerVolumeSpecName "kube-api-access-lf99f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:45.488968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.488938 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf99f\" (UniqueName: \"kubernetes.io/projected/58ca22f7-106e-49f0-bbaf-b11b29145d79-kube-api-access-lf99f\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:45.488968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.488970 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:45.489205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.488984 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca22f7-106e-49f0-bbaf-b11b29145d79-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:45.850431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:45.850388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:46.118530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.116618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:46.181470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.180885 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7"] Nov 01 22:27:46.208466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.201332 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" Nov 01 22:27:46.208466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.202510 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2" event={"ID":"58ca22f7-106e-49f0-bbaf-b11b29145d79","Type":"ContainerDied","Data":"be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a"} Nov 01 22:27:46.208466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.202602 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6a1db562a638f63120b2037eef47e2bfa9699263cbe7e4839b5e8c7c16981a" Nov 01 22:27:46.330325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.330290 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-533654c4-n4pv7"] Nov 01 22:27:46.360466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.360370 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:27:46.361272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361248 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1f688b46-aa57-47f7-b25a-c4934dbc1c92" containerName="revoke-keys" Nov 01 22:27:46.361272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361272 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f688b46-aa57-47f7-b25a-c4934dbc1c92" containerName="revoke-keys" Nov 01 22:27:46.361492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361292 5036 cpu_manager.go:414] "containerMap: removing container" podUID="58ca22f7-106e-49f0-bbaf-b11b29145d79" containerName="revoke-keys" Nov 01 22:27:46.361492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361302 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ca22f7-106e-49f0-bbaf-b11b29145d79" containerName="revoke-keys" Nov 01 22:27:46.361492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361462 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f688b46-aa57-47f7-b25a-c4934dbc1c92" containerName="revoke-keys" Nov 01 22:27:46.361492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.361476 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ca22f7-106e-49f0-bbaf-b11b29145d79" containerName="revoke-keys" Nov 01 22:27:46.366145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.365952 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.374359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.374250 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d2832c-openshift-default-dockercfg-vhkf4\"" Nov 01 22:27:46.381491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.381464 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:27:46.414097 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:46.414071 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ca22f7_106e_49f0_bbaf_b11b29145d79.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:27:46.448478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.446553 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24eccbe5-cd28-4979-b9aa-2e2112acc2a4" path="/var/lib/kubelet/pods/24eccbe5-cd28-4979-b9aa-2e2112acc2a4/volumes" Nov 01 22:27:46.469807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.469764 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.469971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.469883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.469971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.469937 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470037 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470069 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470197 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470329 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470394 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.470508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.470496 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bkw\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.517116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.517084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.575606 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.575765 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.575936 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.575989 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bkw\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.576099 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.576156 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.576249 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.576345 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.576469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.576378 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.578398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.578010 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.579279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.579074 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.579517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.579408 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.579985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.579968 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.580508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.580486 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.582042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.581727 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.585909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.583308 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.591670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.591407 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bkw\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.591670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.591659 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.621022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.620997 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:46.677621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.677539 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\") pod \"19a2881b-6f4c-4b4a-98ac-86399cddff02\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " Nov 01 22:27:46.677772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.677705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\" (UniqueName: \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\") pod \"19a2881b-6f4c-4b4a-98ac-86399cddff02\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " Nov 01 22:27:46.677838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.677803 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\") pod \"19a2881b-6f4c-4b4a-98ac-86399cddff02\" (UID: \"19a2881b-6f4c-4b4a-98ac-86399cddff02\") " Nov 01 22:27:46.677959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.677929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca" pod "19a2881b-6f4c-4b4a-98ac-86399cddff02" (UID: "19a2881b-6f4c-4b4a-98ac-86399cddff02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:46.678542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.678513 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:46.679874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.679851 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k" pod "19a2881b-6f4c-4b4a-98ac-86399cddff02" (UID: "19a2881b-6f4c-4b4a-98ac-86399cddff02"). InnerVolumeSpecName "kube-api-access-96x7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:46.679975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.679911 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca" pod "19a2881b-6f4c-4b4a-98ac-86399cddff02" (UID: "19a2881b-6f4c-4b4a-98ac-86399cddff02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:46.691130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.691104 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:46.780895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.780862 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96x7k\" (UniqueName: \"kubernetes.io/projected/19a2881b-6f4c-4b4a-98ac-86399cddff02-kube-api-access-96x7k\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:46.781037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.780900 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a2881b-6f4c-4b4a-98ac-86399cddff02-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:46.997228 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:46.997198 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6320fc_af60_4568_b57b_2efa090dc1fe.slice/crio-e71cdc119f4e290bd9f8911f9b3c9e382505245f7ad88d753e3f699d8c00bc7e WatchSource:0}: Error finding container e71cdc119f4e290bd9f8911f9b3c9e382505245f7ad88d753e3f699d8c00bc7e: Status 404 returned error can't find the container with id e71cdc119f4e290bd9f8911f9b3c9e382505245f7ad88d753e3f699d8c00bc7e Nov 01 22:27:46.998806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:46.998779 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:27:47.000692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.000661 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:47.000785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.000726 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:47.000785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.000755 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:47.218180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.218141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" event={"ID":"bf6320fc-af60-4568-b57b-2efa090dc1fe","Type":"ContainerStarted","Data":"4dfcd7fb817b46c862f72f71aca57e893fb80f3efa925bca416570535e103432"} Nov 01 22:27:47.218382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.218198 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" event={"ID":"bf6320fc-af60-4568-b57b-2efa090dc1fe","Type":"ContainerStarted","Data":"e71cdc119f4e290bd9f8911f9b3c9e382505245f7ad88d753e3f699d8c00bc7e"} Nov 01 22:27:47.220091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.220064 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" Nov 01 22:27:47.220244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.220070 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9" event={"ID":"19a2881b-6f4c-4b4a-98ac-86399cddff02","Type":"ContainerDied","Data":"19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c"} Nov 01 22:27:47.220244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.220183 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f9df7599c45971d571b09444cd7e3ae9e7f675694f492cbc013f3332bd438c" Nov 01 22:27:47.245884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.245799 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podStartSLOduration=1.245786888 podStartE2EDuration="1.245786888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:47.24277662 +0000 UTC m=+2620.935757107" watchObservedRunningTime="2026-11-01 22:27:47.245786888 +0000 UTC m=+2620.938767431" Nov 01 22:27:47.255967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.255945 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:47.692576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:47.692535 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:48.025848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.025803 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:48.195187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.195135 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:48.517890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.517849 5036 generic.go:356] "Generic (PLEG): container finished" podID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerID="06a46c50e7c6bbc5a12f32312687ea50fe4c1e35373bc22debd55268f618fbc2" exitCode=0 Nov 01 22:27:48.518509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.518476 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerDied","Data":"06a46c50e7c6bbc5a12f32312687ea50fe4c1e35373bc22debd55268f618fbc2"} Nov 01 22:27:48.518627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.518555 5036 scope.go:122] "RemoveContainer" containerID="d661453ba9a870dc2e9dd93b3293b9e60a244afd9eb8b3b272b74979f8bcb578" Nov 01 22:27:48.625727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.625697 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:27:48.651506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651476 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651614 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651649 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651729 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.651976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.651876 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.652149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.652001 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.652149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.652057 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\") pod \"1637c6be-f67c-43ae-b5eb-cf893e79866e\" (UID: \"1637c6be-f67c-43ae-b5eb-cf893e79866e\") " Nov 01 22:27:48.653701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.653603 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:48.654309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.653638 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:48.654600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.653721 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:48.654600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.653882 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:48.657525 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.657500 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:48.657918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.657878 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "kube-api-access-zstzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:48.658916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.658891 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:48.660495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.660439 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:27:48.661348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.661328 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy" pod "1637c6be-f67c-43ae-b5eb-cf893e79866e" (UID: "1637c6be-f67c-43ae-b5eb-cf893e79866e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:48.711674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.711574 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:48.711674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.711655 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:48.731705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.731675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755146 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1637c6be-f67c-43ae-b5eb-cf893e79866e-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755187 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755202 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstzw\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-kube-api-access-zstzw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755220 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755236 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755249 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755262 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755275 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1637c6be-f67c-43ae-b5eb-cf893e79866e-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.760503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:48.755291 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1637c6be-f67c-43ae-b5eb-cf893e79866e-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:48.905927 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:27:48.905884 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:27:49.425999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.425946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:49.546483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.546437 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" Nov 01 22:27:49.547827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.547795 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt" event={"ID":"1637c6be-f67c-43ae-b5eb-cf893e79866e","Type":"ContainerDied","Data":"759aa5d3718ac9d5320d952930bbf4f01964e309bbd3771f06901bc9d8636c9d"} Nov 01 22:27:49.547903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.547862 5036 scope.go:122] "RemoveContainer" containerID="06a46c50e7c6bbc5a12f32312687ea50fe4c1e35373bc22debd55268f618fbc2" Nov 01 22:27:49.599557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.590490 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:27:49.606984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.606956 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-s9gmt"] Nov 01 22:27:49.695924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.695835 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:49.696198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:49.696166 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:50.212469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:50.204205 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:50.441353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:50.441314 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" path="/var/lib/kubelet/pods/1637c6be-f67c-43ae-b5eb-cf893e79866e/volumes" Nov 01 22:27:50.692615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:50.692578 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:50.692916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:50.692867 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:50.920310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:50.920274 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:51.423059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423020 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw"] Nov 01 22:27:51.423718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423702 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:51.423769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423719 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:51.423769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423742 5036 cpu_manager.go:414] "containerMap: removing container" podUID="19a2881b-6f4c-4b4a-98ac-86399cddff02" containerName="revoke-keys" Nov 01 22:27:51.423769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423747 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a2881b-6f4c-4b4a-98ac-86399cddff02" containerName="revoke-keys" Nov 01 22:27:51.423875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423778 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:51.423875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423783 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:51.423938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423901 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:51.423938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.423913 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a2881b-6f4c-4b4a-98ac-86399cddff02" containerName="revoke-keys" Nov 01 22:27:51.429065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.429040 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.433260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.433240 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:27:51.434484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.434435 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:27:51.436493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.436427 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw"] Nov 01 22:27:51.534024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.533987 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcmhr\" (UniqueName: \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.534184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.534036 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.534265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.534194 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.636911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.636867 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcmhr\" (UniqueName: \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.637363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.636990 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.637560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.637527 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.638063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.638036 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.640965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.640367 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.651023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.650998 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcmhr\" (UniqueName: \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\") pod \"maas-api-revoke-sub-579dcf8f-wdlzw\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:51.705531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.701944 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:51.705531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.702032 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:51.705531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.704265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:51.756108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:51.753903 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:52.097474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.095667 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw"] Nov 01 22:27:52.112827 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:27:52.112793 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a7df3_9338_4dc6_a9bf_a16d6594d9fa.slice/crio-aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1 WatchSource:0}: Error finding container aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1: Status 404 returned error can't find the container with id aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1 Nov 01 22:27:52.261768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.261666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:52.424213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.424112 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:52.592467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.591316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:52.679408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.679320 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc"] Nov 01 22:27:52.695686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.695229 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:52.695686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.695286 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:52.703686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.703639 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-da34ca-z65qc"] Nov 01 22:27:52.713177 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.712893 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" event={"ID":"575a7df3-9338-4dc6-a9bf-a16d6594d9fa","Type":"ContainerStarted","Data":"aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1"} Nov 01 22:27:52.808000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:52.807967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:53.020430 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.020390 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:53.160740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.158562 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:27:53.166478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.163055 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1637c6be-f67c-43ae-b5eb-cf893e79866e" containerName="istio-proxy" Nov 01 22:27:53.167878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.167852 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:27:53.168035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.167976 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.317727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.317684 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.317923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.317898 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318051 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rrz\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318113 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318183 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318221 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318252 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318382 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.318494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.318473 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420165 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420331 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420413 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rrz\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420478 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420510 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420544 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420575 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420638 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.420733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.420697 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.422161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.422107 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.422284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.422239 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.422366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.422295 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.422759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.422733 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.422932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.422860 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.423609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.423584 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.423896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.423861 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.434142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.434113 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.434498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.434437 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rrz\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.516780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.516726 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:53.583580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.583545 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:53.693220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.692744 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 01 22:27:53.693220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.692821 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 01 22:27:53.727307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.727216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:53.728489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.728455 5036 generic.go:356] "Generic (PLEG): container finished" podID="575a7df3-9338-4dc6-a9bf-a16d6594d9fa" containerID="e28c8b07566dcc9e370053c170fdda0fdff9397a58e86e6b49dff0d65bc80724" exitCode=0 Nov 01 22:27:53.729379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.728515 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" event={"ID":"575a7df3-9338-4dc6-a9bf-a16d6594d9fa","Type":"ContainerDied","Data":"e28c8b07566dcc9e370053c170fdda0fdff9397a58e86e6b49dff0d65bc80724"} Nov 01 22:27:53.887544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:53.885410 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:54.048983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.048012 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:54.048983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.048100 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:54.048983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.048140 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:54.049233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.049208 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:27:54.443993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.443899 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7522fd75-37df-410a-b235-ea3f3a91c48b" path="/var/lib/kubelet/pods/7522fd75-37df-410a-b235-ea3f3a91c48b/volumes" Nov 01 22:27:54.701917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.701809 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:54.705564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.705492 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:54.705781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.703714 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:54.707076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.707055 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:27:54.739806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.739764 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" event={"ID":"61197dc0-cd57-4275-80a4-8a72b298d957","Type":"ContainerStarted","Data":"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0"} Nov 01 22:27:54.739962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.739828 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" event={"ID":"61197dc0-cd57-4275-80a4-8a72b298d957","Type":"ContainerStarted","Data":"851cc91fcd0807e16dcad35e1a4d7af4fce2f1ab5abfe11f2e35d9ac6d8d5093"} Nov 01 22:27:54.905479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.898191 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podStartSLOduration=1.898164183 podStartE2EDuration="1.898164183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.83662407 +0000 UTC m=+2628.529604558" watchObservedRunningTime="2026-11-01 22:27:54.898164183 +0000 UTC m=+2628.591144670" Nov 01 22:27:54.930492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:54.930428 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:55.157468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.156978 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:55.186539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.159487 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:27:55.436798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.436234 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:27:55.440737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.437384 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.513469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.510602 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-db715b-openshift-default-dockercfg-w26zq\"" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.558768 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.558876 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.558921 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559130 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559272 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559373 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559403 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559513 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbpq\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.559679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.559546 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.581061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.581026 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:27:55.581342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.581313 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:27:55.582261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.581761 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.661733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.661795 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.661918 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662002 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662070 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662171 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbpq\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662198 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662306 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.662807 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.663055 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.663282 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.663545 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.665462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.665109 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.670100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.668661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.673290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.673244 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.700160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.683322 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbpq\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.722105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.711987 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\") pod \"e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:55.742009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.741979 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:55.750238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.750211 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" Nov 01 22:27:55.750360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.750246 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw" event={"ID":"575a7df3-9338-4dc6-a9bf-a16d6594d9fa","Type":"ContainerDied","Data":"aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1"} Nov 01 22:27:55.750360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.750291 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aafba3f0085df03e549ba1b680e84a8a1e76d1595b7333d38fae51098efb07e1" Nov 01 22:27:55.770683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.770651 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\") pod \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " Nov 01 22:27:55.770831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.770743 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\" (UniqueName: \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\") pod \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " Nov 01 22:27:55.770879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.770855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\") pod \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\" (UID: \"575a7df3-9338-4dc6-a9bf-a16d6594d9fa\") " Nov 01 22:27:55.772226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.772200 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca" pod "575a7df3-9338-4dc6-a9bf-a16d6594d9fa" (UID: "575a7df3-9338-4dc6-a9bf-a16d6594d9fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:55.773893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.773431 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:55.774671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.774639 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr" pod "575a7df3-9338-4dc6-a9bf-a16d6594d9fa" (UID: "575a7df3-9338-4dc6-a9bf-a16d6594d9fa"). InnerVolumeSpecName "kube-api-access-qcmhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:55.781584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.781537 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca" pod "575a7df3-9338-4dc6-a9bf-a16d6594d9fa" (UID: "575a7df3-9338-4dc6-a9bf-a16d6594d9fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:55.904479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.896041 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:55.904479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.896085 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcmhr\" (UniqueName: \"kubernetes.io/projected/575a7df3-9338-4dc6-a9bf-a16d6594d9fa-kube-api-access-qcmhr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:55.919925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.919794 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:55.943800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:55.943758 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:56.293377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.293228 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:27:56.316521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.306847 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:56.316521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.306945 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:56.316521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.306989 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:27:56.339044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.339015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:56.526849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.526817 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:27:56.528562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.528522 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:27:56.767404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.767368 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerStarted","Data":"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2"} Nov 01 22:27:56.767572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.767430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerStarted","Data":"36d44bd63a584fc70d63d01a3db5d13e8bb15b75cbcc9376f1fbef305e2e66da"} Nov 01 22:27:56.833985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.833854 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podStartSLOduration=1.833833839 podStartE2EDuration="1.833833839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:56.793399135 +0000 UTC m=+2630.486379625" watchObservedRunningTime="2026-11-01 22:27:56.833833839 +0000 UTC m=+2630.526814327" Nov 01 22:27:56.946583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:56.946533 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:57.519910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:57.519874 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:27:57.520199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:57.520172 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:27:57.661467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:57.660006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:57.951778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:57.951693 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:58.519959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:58.519924 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:27:58.520144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:58.519976 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:27:58.817512 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:58.817408 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:58.818413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:58.818383 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:27:59.021569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.021520 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:59.154691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.154586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:59.314410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.314374 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:59.521769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.520893 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:27:59.521769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.520953 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:27:59.934209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:27:59.934109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.061507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.061436 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.314465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.311322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.527465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.527394 5036 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 01 22:28:00.527659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.527475 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 01 22:28:00.531464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.528881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.782490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.779415 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.903766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.903716 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:00.967584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.967537 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t"] Nov 01 22:28:00.968705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.968681 5036 cpu_manager.go:414] "containerMap: removing container" podUID="575a7df3-9338-4dc6-a9bf-a16d6594d9fa" containerName="revoke-keys" Nov 01 22:28:00.968772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.968710 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a7df3-9338-4dc6-a9bf-a16d6594d9fa" containerName="revoke-keys" Nov 01 22:28:00.968968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.968953 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a7df3-9338-4dc6-a9bf-a16d6594d9fa" containerName="revoke-keys" Nov 01 22:28:00.974856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.974829 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:00.977426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.977399 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t"] Nov 01 22:28:00.979467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.979430 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:28:00.980807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:00.980784 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:28:01.126009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.125908 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.126437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.126380 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4z7b\" (UniqueName: \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.126953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.126930 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.128723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.128571 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:01.230080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.230037 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.230253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.230107 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4z7b\" (UniqueName: \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.230338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.230254 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.237491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.237109 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.238226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.238199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.249864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.249826 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4z7b\" (UniqueName: \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.276381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.276017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:01.321665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.321634 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:01.792691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.782747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:28:01.792691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.784040 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:28:01.792691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:01.786173 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:28:02.051213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.051179 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t"] Nov 01 22:28:02.054492 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:02.054439 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice/crio-090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b WatchSource:0}: Error finding container 090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b: Status 404 returned error can't find the container with id 090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b Nov 01 22:28:02.215946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.215897 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf"] Nov 01 22:28:02.228064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.227844 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.228261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.228234 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf"] Nov 01 22:28:02.296338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.296299 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z575h\" (UniqueName: \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.296548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.296530 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.296615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.296601 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.332756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.332682 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:02.399013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.398971 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.399226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.399062 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.399289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.399252 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z575h\" (UniqueName: \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.399912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.399873 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.403585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.403485 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.423616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.423581 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z575h\" (UniqueName: \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\") pod \"maas-api-revoke-sub-c629600c-s96nf\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.570193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.570159 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:02.887566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.884427 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" event={"ID":"40e84511-dd47-4e6c-8866-5cfbebf1d415","Type":"ContainerStarted","Data":"6072dea5a025783a1724c8ac09e2599c0893204ae9f86cf6d353700c1f2df316"} Nov 01 22:28:02.887566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.884551 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" event={"ID":"40e84511-dd47-4e6c-8866-5cfbebf1d415","Type":"ContainerStarted","Data":"090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b"} Nov 01 22:28:02.889242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.889210 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf"] Nov 01 22:28:02.889401 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:02.889367 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6da6937_d872_47db_92a8_abfef5ae679b.slice/crio-76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65 WatchSource:0}: Error finding container 76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65: Status 404 returned error can't find the container with id 76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65 Nov 01 22:28:02.910783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:02.910716 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" podStartSLOduration=2.910698327 podStartE2EDuration="2.910698327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:02.903307791 +0000 UTC m=+2636.596288274" watchObservedRunningTime="2026-11-01 22:28:02.910698327 +0000 UTC m=+2636.603678814" Nov 01 22:28:03.338154 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.338118 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:03.636977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.636934 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:28:03.643368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.643136 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.648556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.648524 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-09656e\"/\"default-dockercfg-6rzx7\"" Nov 01 22:28:03.651134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.651108 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-09656e\"/\"ipp-route-09656e-kserve-self-signed-certs\"" Nov 01 22:28:03.661778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.661748 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-09656e\"/\"kube-root-ca.crt\"" Nov 01 22:28:03.663388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.662538 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-09656e\"/\"openshift-service-ca.crt\"" Nov 01 22:28:03.669886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.669860 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:28:03.711479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.708541 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:03.739029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.738993 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.739231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.739135 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.739231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.739181 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.739353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.739337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtvr\" (UniqueName: \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.739420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.739386 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.739506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.739460 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.865655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.865615 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtvr\" (UniqueName: \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.865946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.865916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.866159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.866120 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.866364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.866345 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.890757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.890675 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.890994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.890975 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.920465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.920095 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.920465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.884975 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.920465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.885261 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.926566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.926493 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtvr\" (UniqueName: \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.936524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.927777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.938802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.938777 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\") pod \"ipp-route-09656e-kserve-556984d957-gv6vt\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:03.984469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:03.983168 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:04.063097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.063054 5036 generic.go:356] "Generic (PLEG): container finished" podID="40e84511-dd47-4e6c-8866-5cfbebf1d415" containerID="6072dea5a025783a1724c8ac09e2599c0893204ae9f86cf6d353700c1f2df316" exitCode=0 Nov 01 22:28:04.063413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.063385 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" event={"ID":"40e84511-dd47-4e6c-8866-5cfbebf1d415","Type":"ContainerDied","Data":"6072dea5a025783a1724c8ac09e2599c0893204ae9f86cf6d353700c1f2df316"} Nov 01 22:28:04.074487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.072920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" event={"ID":"e6da6937-d872-47db-92a8-abfef5ae679b","Type":"ContainerStarted","Data":"b2ab58c78fab0ed49a0a1b139dd46984aedada5d89f3be65448284dcf01a6768"} Nov 01 22:28:04.074487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.072997 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" event={"ID":"e6da6937-d872-47db-92a8-abfef5ae679b","Type":"ContainerStarted","Data":"76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65"} Nov 01 22:28:04.129399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.129335 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" podStartSLOduration=2.129312218 podStartE2EDuration="2.129312218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:04.113059946 +0000 UTC m=+2637.806040432" watchObservedRunningTime="2026-11-01 22:28:04.129312218 +0000 UTC m=+2637.822292704" Nov 01 22:28:04.329980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.329913 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:04.447995 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:04.447961 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc834ace4_3421_4ae4_8b0f_3e9c55543c35.slice/crio-f599aa343f6d1b9adfa47b5cda15d7a2167a189197dbe6b23a435384ba2e0129 WatchSource:0}: Error finding container f599aa343f6d1b9adfa47b5cda15d7a2167a189197dbe6b23a435384ba2e0129: Status 404 returned error can't find the container with id f599aa343f6d1b9adfa47b5cda15d7a2167a189197dbe6b23a435384ba2e0129 Nov 01 22:28:04.450775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:04.450751 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:28:05.086393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.086352 5036 generic.go:356] "Generic (PLEG): container finished" podID="e6da6937-d872-47db-92a8-abfef5ae679b" containerID="b2ab58c78fab0ed49a0a1b139dd46984aedada5d89f3be65448284dcf01a6768" exitCode=0 Nov 01 22:28:05.086630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.086520 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" event={"ID":"e6da6937-d872-47db-92a8-abfef5ae679b","Type":"ContainerDied","Data":"b2ab58c78fab0ed49a0a1b139dd46984aedada5d89f3be65448284dcf01a6768"} Nov 01 22:28:05.087413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.087388 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:05.090243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.090213 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" event={"ID":"c834ace4-3421-4ae4-8b0f-3e9c55543c35","Type":"ContainerStarted","Data":"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26"} Nov 01 22:28:05.090410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.090393 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" event={"ID":"c834ace4-3421-4ae4-8b0f-3e9c55543c35","Type":"ContainerStarted","Data":"f599aa343f6d1b9adfa47b5cda15d7a2167a189197dbe6b23a435384ba2e0129"} Nov 01 22:28:05.204632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.204562 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" podStartSLOduration=2.204541597 podStartE2EDuration="2.204541597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:05.201791872 +0000 UTC m=+2638.894772358" watchObservedRunningTime="2026-11-01 22:28:05.204541597 +0000 UTC m=+2638.897522081" Nov 01 22:28:05.461556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.461524 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:05.542625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.542578 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\") pod \"40e84511-dd47-4e6c-8866-5cfbebf1d415\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " Nov 01 22:28:05.542803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.542705 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\") pod \"40e84511-dd47-4e6c-8866-5cfbebf1d415\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " Nov 01 22:28:05.542937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.542916 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\" (UniqueName: \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\") pod \"40e84511-dd47-4e6c-8866-5cfbebf1d415\" (UID: \"40e84511-dd47-4e6c-8866-5cfbebf1d415\") " Nov 01 22:28:05.543148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.543007 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca" pod "40e84511-dd47-4e6c-8866-5cfbebf1d415" (UID: "40e84511-dd47-4e6c-8866-5cfbebf1d415"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:05.544297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.543766 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.545721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.545692 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca" pod "40e84511-dd47-4e6c-8866-5cfbebf1d415" (UID: "40e84511-dd47-4e6c-8866-5cfbebf1d415"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:05.546006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.545973 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b" pod "40e84511-dd47-4e6c-8866-5cfbebf1d415" (UID: "40e84511-dd47-4e6c-8866-5cfbebf1d415"). InnerVolumeSpecName "kube-api-access-r4z7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:05.646145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.646111 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4z7b\" (UniqueName: \"kubernetes.io/projected/40e84511-dd47-4e6c-8866-5cfbebf1d415-kube-api-access-r4z7b\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.646145 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.646143 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e84511-dd47-4e6c-8866-5cfbebf1d415-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.711919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.711818 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 22:28:05.734714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.734675 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:28:05.734955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.734927 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" podUID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" containerName="maas-api" containerID="cri-o://2e760f8d8019d86eb4eb66bd8c36247e41b16a2718f3ee0ec0a0d1f2a0c398d2" gracePeriod=30 Nov 01 22:28:05.802067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.801986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:05.935843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:05.935793 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:06.037481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.001077 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:06.553818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.553750 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:06.637878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.637845 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" Nov 01 22:28:06.654335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.654295 5036 generic.go:356] "Generic (PLEG): container finished" podID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" containerID="2e760f8d8019d86eb4eb66bd8c36247e41b16a2718f3ee0ec0a0d1f2a0c398d2" exitCode=0 Nov 01 22:28:06.659645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.659486 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2"] Nov 01 22:28:06.659645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.659531 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0bb375-nxfs2"] Nov 01 22:28:06.659645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.659557 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t" event={"ID":"40e84511-dd47-4e6c-8866-5cfbebf1d415","Type":"ContainerDied","Data":"090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b"} Nov 01 22:28:06.659645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.659578 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090090c5418981b59f04e97d6d28ad91c211a6caa835a424009f618f3557ed7b" Nov 01 22:28:06.659645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.659591 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" event={"ID":"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd","Type":"ContainerDied","Data":"2e760f8d8019d86eb4eb66bd8c36247e41b16a2718f3ee0ec0a0d1f2a0c398d2"} Nov 01 22:28:06.667095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.667062 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:28:06.684356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.684316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:06.773127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.773093 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\") pod \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " Nov 01 22:28:06.773288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.773161 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\") pod \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " Nov 01 22:28:06.773288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.773280 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\") pod \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " Nov 01 22:28:06.773379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.773325 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\" (UniqueName: \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\") pod \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\" (UID: \"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd\") " Nov 01 22:28:06.773902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.773870 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca" pod "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" (UID: "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:06.774426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.774402 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.776913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.776882 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl" pod "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" (UID: "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd"). InnerVolumeSpecName "kube-api-access-4wdpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:06.777848 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:06.777824 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:06.788958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.787791 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls" pod "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" (UID: "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:06.789362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.789339 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs" pod "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" (UID: "7ae66d49-8f56-4e7b-844d-5f3c6c9724cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:06.802749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.802725 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:06.875934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.875896 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.875934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.875935 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.876190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.875949 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wdpl\" (UniqueName: \"kubernetes.io/projected/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd-kube-api-access-4wdpl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.905192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.905168 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:06.977372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.977336 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\" (UniqueName: \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\") pod \"e6da6937-d872-47db-92a8-abfef5ae679b\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " Nov 01 22:28:06.977617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.977597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\") pod \"e6da6937-d872-47db-92a8-abfef5ae679b\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " Nov 01 22:28:06.977744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.977726 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\") pod \"e6da6937-d872-47db-92a8-abfef5ae679b\" (UID: \"e6da6937-d872-47db-92a8-abfef5ae679b\") " Nov 01 22:28:06.978025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.977998 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca" pod "e6da6937-d872-47db-92a8-abfef5ae679b" (UID: "e6da6937-d872-47db-92a8-abfef5ae679b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:06.978486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.978423 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.980150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.980122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca" pod "e6da6937-d872-47db-92a8-abfef5ae679b" (UID: "e6da6937-d872-47db-92a8-abfef5ae679b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:06.980250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:06.980223 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h" pod "e6da6937-d872-47db-92a8-abfef5ae679b" (UID: "e6da6937-d872-47db-92a8-abfef5ae679b"). InnerVolumeSpecName "kube-api-access-z575h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:07.080454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.080355 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6da6937-d872-47db-92a8-abfef5ae679b-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:07.080454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.080388 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z575h\" (UniqueName: \"kubernetes.io/projected/e6da6937-d872-47db-92a8-abfef5ae679b-kube-api-access-z575h\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:07.424615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.424530 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:07.664756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.664725 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" Nov 01 22:28:07.665224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.664756 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57" event={"ID":"7ae66d49-8f56-4e7b-844d-5f3c6c9724cd","Type":"ContainerDied","Data":"932617b1fea871c6cff421bb845ea9006583ff68b5e4332478f067da06ebd6d0"} Nov 01 22:28:07.665224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.664830 5036 scope.go:122] "RemoveContainer" containerID="2e760f8d8019d86eb4eb66bd8c36247e41b16a2718f3ee0ec0a0d1f2a0c398d2" Nov 01 22:28:07.666637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.666612 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" Nov 01 22:28:07.666759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.666647 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf" event={"ID":"e6da6937-d872-47db-92a8-abfef5ae679b","Type":"ContainerDied","Data":"76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65"} Nov 01 22:28:07.666759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.666694 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a943bde5cac11c8d9647aa67cae77b93383c9e6bc4f8ab20cc012dc869cb65" Nov 01 22:28:07.738275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.738246 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:28:07.760753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:07.760727 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0bb375-8c66fb4bf-5dl57"] Nov 01 22:28:08.394206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:08.394170 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:08.508942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:08.508903 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ca22f7-106e-49f0-bbaf-b11b29145d79" path="/var/lib/kubelet/pods/58ca22f7-106e-49f0-bbaf-b11b29145d79/volumes" Nov 01 22:28:08.509543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:08.509521 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" path="/var/lib/kubelet/pods/7ae66d49-8f56-4e7b-844d-5f3c6c9724cd/volumes" Nov 01 22:28:09.009508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.007298 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:09.164468 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:09.156076 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:09.182476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.182320 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:09.352469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.347680 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:09.398465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.397337 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9"] Nov 01 22:28:09.513017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.512978 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-de0cba-bpjc9"] Nov 01 22:28:09.573199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.573164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:09.816813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:09.816772 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:10.341888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:10.341849 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:10.444955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:10.444918 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a2881b-6f4c-4b4a-98ac-86399cddff02" path="/var/lib/kubelet/pods/19a2881b-6f4c-4b4a-98ac-86399cddff02/volumes" Nov 01 22:28:11.142872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:11.142829 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 22:28:11.196524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:11.195462 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:11.320949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:11.320909 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:12.121411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:12.121360 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:12.922184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:12.922135 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:13.275820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:13.275785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:13.405333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:13.405293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:13.596504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:13.596367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:13.740118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:13.740030 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:13.984286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:13.984202 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:14.001835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.001806 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:14.020291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.020261 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f"] Nov 01 22:28:14.021047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021026 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" containerName="maas-api" Nov 01 22:28:14.021155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021047 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" containerName="maas-api" Nov 01 22:28:14.021155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021083 5036 cpu_manager.go:414] "containerMap: removing container" podUID="40e84511-dd47-4e6c-8866-5cfbebf1d415" containerName="revoke-keys" Nov 01 22:28:14.021155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021091 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e84511-dd47-4e6c-8866-5cfbebf1d415" containerName="revoke-keys" Nov 01 22:28:14.021155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021118 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e6da6937-d872-47db-92a8-abfef5ae679b" containerName="revoke-keys" Nov 01 22:28:14.021155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021125 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6da6937-d872-47db-92a8-abfef5ae679b" containerName="revoke-keys" Nov 01 22:28:14.021541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021299 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae66d49-8f56-4e7b-844d-5f3c6c9724cd" containerName="maas-api" Nov 01 22:28:14.021541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021317 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e84511-dd47-4e6c-8866-5cfbebf1d415" containerName="revoke-keys" Nov 01 22:28:14.021541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.021330 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6da6937-d872-47db-92a8-abfef5ae679b" containerName="revoke-keys" Nov 01 22:28:14.032954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.032926 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.049407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.049379 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f"] Nov 01 22:28:14.116199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.116154 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.116669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.116633 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftr9z\" (UniqueName: \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.117068 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.117046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.132942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.132911 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:28:14.133049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.133014 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:28:14.139490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.137087 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:14.311466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.305899 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr9z\" (UniqueName: \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.311466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.306097 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.311466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.306243 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.334470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.332496 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.341699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.340294 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.362474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.362423 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr9z\" (UniqueName: \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.390297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.390266 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:14.725865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.725833 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f"] Nov 01 22:28:14.726213 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:14.726190 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7267a8_8175_48a0_8bd2_fdf2251d11e9.slice/crio-40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25 WatchSource:0}: Error finding container 40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25: Status 404 returned error can't find the container with id 40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25 Nov 01 22:28:14.848849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.848813 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" event={"ID":"7b7267a8-8175-48a0-8bd2-fdf2251d11e9","Type":"ContainerStarted","Data":"40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25"} Nov 01 22:28:14.861008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.860982 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:28:14.913464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:14.911267 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:15.838986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:15.838941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:15.929353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:15.926097 5036 generic.go:356] "Generic (PLEG): container finished" podID="7b7267a8-8175-48a0-8bd2-fdf2251d11e9" containerID="98dce5c98798a35383f50e3a1801bdec043178fc306c0016981c6effaab81bce" exitCode=0 Nov 01 22:28:15.929353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:15.926143 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" event={"ID":"7b7267a8-8175-48a0-8bd2-fdf2251d11e9","Type":"ContainerDied","Data":"98dce5c98798a35383f50e3a1801bdec043178fc306c0016981c6effaab81bce"} Nov 01 22:28:16.143310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.143261 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" containerID="cri-o://1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c" gracePeriod=25 Nov 01 22:28:16.160347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.160317 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:16.290699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.290664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:16.451462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.451413 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:28:16.451670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.451622 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" containerID="cri-o://53744ef78a182040ea3d717173c14dde2a5b4971a52b8db418d47a529ddba18b" gracePeriod=30 Nov 01 22:28:16.481090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.480982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:16.608589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.608437 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:16.934550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:16.930839 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:17.048467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.042908 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:28:17.277523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.272053 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6"] Nov 01 22:28:17.277523 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:17.275501 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:17.291530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.290485 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6"] Nov 01 22:28:17.291530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.290637 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.424889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.424841 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.425079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.424975 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.425379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.425350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bggt\" (UniqueName: \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.515026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.514990 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:17.527083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.527038 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bggt\" (UniqueName: \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.527328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.527307 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.527427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.527373 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.529401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.529335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.536431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.535129 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.554021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.553992 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bggt\" (UniqueName: \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\") pod \"maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.626671 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.626638 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:17.629128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.629095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\") pod \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " Nov 01 22:28:17.629280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.629199 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\" (UniqueName: \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\") pod \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " Nov 01 22:28:17.629463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.629419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\") pod \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\" (UID: \"7b7267a8-8175-48a0-8bd2-fdf2251d11e9\") " Nov 01 22:28:17.629592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.629569 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca" pod "7b7267a8-8175-48a0-8bd2-fdf2251d11e9" (UID: "7b7267a8-8175-48a0-8bd2-fdf2251d11e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:17.630142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.630114 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:17.631612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.631587 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca" pod "7b7267a8-8175-48a0-8bd2-fdf2251d11e9" (UID: "7b7267a8-8175-48a0-8bd2-fdf2251d11e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:17.631939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.631915 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z" pod "7b7267a8-8175-48a0-8bd2-fdf2251d11e9" (UID: "7b7267a8-8175-48a0-8bd2-fdf2251d11e9"). InnerVolumeSpecName "kube-api-access-ftr9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:17.732389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.732353 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:17.732533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.732394 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftr9z\" (UniqueName: \"kubernetes.io/projected/7b7267a8-8175-48a0-8bd2-fdf2251d11e9-kube-api-access-ftr9z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:17.807692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.807585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:17.874290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.874247 5036 scope.go:122] "RemoveContainer" containerID="91bc33acee3041ab600bdb6724b38454dfd90b39eaf83e63954c62480fd398e5" Nov 01 22:28:17.892308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.892283 5036 scope.go:122] "RemoveContainer" containerID="84d5f1555e6aaf049337bce3e14d7baf151104ceec98b8e154f215a3f066ad49" Nov 01 22:28:17.905044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.905014 5036 scope.go:122] "RemoveContainer" containerID="33a368d586fc6a731daa8387c7e2e986c79dbbfc1f1c355a8acb4f7557fecabf" Nov 01 22:28:17.918803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.918700 5036 scope.go:122] "RemoveContainer" containerID="88a74be70b5e9e3c9233b9bf210974659126633e74d6a8e2c4cfde0d5a646079" Nov 01 22:28:17.920744 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:17.920717 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055d8b49_77f8_4e2b_831b_3dd665a7d70e.slice/crio-80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7 WatchSource:0}: Error finding container 80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7: Status 404 returned error can't find the container with id 80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7 Nov 01 22:28:17.920877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.920758 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6"] Nov 01 22:28:17.932649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:17.932626 5036 scope.go:122] "RemoveContainer" containerID="ed5e47822ef660f4da6da6b4b72d5b0941383eac8245b3c5a8b3260091b80da4" Nov 01 22:28:18.049690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.049054 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" Nov 01 22:28:18.049690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.049054 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f" event={"ID":"7b7267a8-8175-48a0-8bd2-fdf2251d11e9","Type":"ContainerDied","Data":"40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25"} Nov 01 22:28:18.049690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.049235 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a21ae554b0c5c3e2288d6232bfab32c22e842283f04e79adb8079d4069ff25" Nov 01 22:28:18.051813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.051786 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" event={"ID":"055d8b49-77f8-4e2b-831b-3dd665a7d70e","Type":"ContainerStarted","Data":"80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7"} Nov 01 22:28:18.764477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.762250 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:18.982507 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:18.980497 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:28:18.999640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:18.996362 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:19.122469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:19.121944 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" event={"ID":"055d8b49-77f8-4e2b-831b-3dd665a7d70e","Type":"ContainerStarted","Data":"32a149741da246ce1870e28a3c6eb99032f59c62e0792ac298766f59493e1daa"} Nov 01 22:28:19.895468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:19.892393 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:20.098469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.087849 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:20.136926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.136890 5036 generic.go:356] "Generic (PLEG): container finished" podID="055d8b49-77f8-4e2b-831b-3dd665a7d70e" containerID="32a149741da246ce1870e28a3c6eb99032f59c62e0792ac298766f59493e1daa" exitCode=0 Nov 01 22:28:20.137411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.136970 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" event={"ID":"055d8b49-77f8-4e2b-831b-3dd665a7d70e","Type":"ContainerDied","Data":"32a149741da246ce1870e28a3c6eb99032f59c62e0792ac298766f59493e1daa"} Nov 01 22:28:20.351255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.349381 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:20.894892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.894846 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" containerID="cri-o://164ea1cd00c9424db37d3765403d9e9b1cf2661e4f669a352c812472187e6e53" gracePeriod=45 Nov 01 22:28:20.911833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.911805 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:20.958257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.958225 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\" (UniqueName: \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\") pod \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " Nov 01 22:28:20.960369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.960340 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\") pod \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " Nov 01 22:28:20.961652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.961627 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\") pod \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\" (UID: \"055d8b49-77f8-4e2b-831b-3dd665a7d70e\") " Nov 01 22:28:20.965032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.965006 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca" pod "055d8b49-77f8-4e2b-831b-3dd665a7d70e" (UID: "055d8b49-77f8-4e2b-831b-3dd665a7d70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:20.973896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.973821 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:20.995766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.995675 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt" pod "055d8b49-77f8-4e2b-831b-3dd665a7d70e" (UID: "055d8b49-77f8-4e2b-831b-3dd665a7d70e"). InnerVolumeSpecName "kube-api-access-6bggt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:20.999013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:20.998971 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca" pod "055d8b49-77f8-4e2b-831b-3dd665a7d70e" (UID: "055d8b49-77f8-4e2b-831b-3dd665a7d70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:21.086330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.084756 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Nov 01 22:28:21.086330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.086236 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.086330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.086270 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055d8b49-77f8-4e2b-831b-3dd665a7d70e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.086330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.086284 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bggt\" (UniqueName: \"kubernetes.io/projected/055d8b49-77f8-4e2b-831b-3dd665a7d70e-kube-api-access-6bggt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.106464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.106361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:21.215472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.212567 5036 generic.go:356] "Generic (PLEG): container finished" podID="813df7b3-d181-4675-a695-af946f32a232" containerID="164ea1cd00c9424db37d3765403d9e9b1cf2661e4f669a352c812472187e6e53" exitCode=0 Nov 01 22:28:21.215472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.212682 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" event={"ID":"813df7b3-d181-4675-a695-af946f32a232","Type":"ContainerDied","Data":"164ea1cd00c9424db37d3765403d9e9b1cf2661e4f669a352c812472187e6e53"} Nov 01 22:28:21.226175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.225328 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" Nov 01 22:28:21.251630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.225171 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6" event={"ID":"055d8b49-77f8-4e2b-831b-3dd665a7d70e","Type":"ContainerDied","Data":"80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7"} Nov 01 22:28:21.251630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.244655 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80156c926934f699bf56c340df423892812b9ccd7596282af422f9c78377fdb7" Nov 01 22:28:21.331929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.331858 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:21.810626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.791731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:21.961523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:21.961243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:22.089775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.089669 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 22:28:22.190984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.173355 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:22.499544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.491007 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.576969 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\" (UniqueName: \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.577230 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.577360 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\" (UniqueName: \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.577818 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.587823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.590469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.587913 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\") pod \"813df7b3-d181-4675-a695-af946f32a232\" (UID: \"813df7b3-d181-4675-a695-af946f32a232\") " Nov 01 22:28:22.597460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.596745 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:22.625462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.625394 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:22.729483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.727659 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:22.737982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.737944 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.737982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.737985 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.738201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.738005 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.755812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.751400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:22.755812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.751522 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:22.757236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.757181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx" pod "813df7b3-d181-4675-a695-af946f32a232" (UID: "813df7b3-d181-4675-a695-af946f32a232"). InnerVolumeSpecName "kube-api-access-pwfhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:22.767626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.767062 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 22:28:22.769829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.768695 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" event={"ID":"813df7b3-d181-4675-a695-af946f32a232","Type":"ContainerDied","Data":"d623482b8e1e210a5aa1a16c4f3f3edcc6bc6c305834638575d3af16fede3d63"} Nov 01 22:28:22.769829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.768750 5036 scope.go:122] "RemoveContainer" containerID="164ea1cd00c9424db37d3765403d9e9b1cf2661e4f669a352c812472187e6e53" Nov 01 22:28:22.769829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.768982 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:28:22.955080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.955042 5036 generic.go:356] "Generic (PLEG): container finished" podID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerID="53744ef78a182040ea3d717173c14dde2a5b4971a52b8db418d47a529ddba18b" exitCode=0 Nov 01 22:28:23.056703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.968532 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfhx\" (UniqueName: \"kubernetes.io/projected/813df7b3-d181-4675-a695-af946f32a232-kube-api-access-pwfhx\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.056921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.056903 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813df7b3-d181-4675-a695-af946f32a232-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.057023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.057009 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813df7b3-d181-4675-a695-af946f32a232-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.057118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.014225 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:28:23.057693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:22.978379 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" event={"ID":"e2476a2b-8eb0-4b13-ae22-47c72902a67d","Type":"ContainerDied","Data":"53744ef78a182040ea3d717173c14dde2a5b4971a52b8db418d47a529ddba18b"} Nov 01 22:28:23.089478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.089089 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 01 22:28:23.089772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.089735 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:28:23.090207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.089804 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:28:23.090413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.090395 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h" Nov 01 22:28:23.104885 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.104855 5036 scope.go:122] "RemoveContainer" containerID="53744ef78a182040ea3d717173c14dde2a5b4971a52b8db418d47a529ddba18b" Nov 01 22:28:23.110813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.110786 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.162569 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.162671 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.162790 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.162935 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.163020 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.163062 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.163091 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.163378 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.164465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.163422 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\") pod \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\" (UID: \"e2476a2b-8eb0-4b13-ae22-47c72902a67d\") " Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.168918 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.169345 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.169695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.170135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.170281 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:23.181479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.170631 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:23.273387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.221203 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "kube-api-access-hsw7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:23.273387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.221997 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:28:23.273387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.222135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:23.273387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.259525 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:23.273387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.260767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token" pod "e2476a2b-8eb0-4b13-ae22-47c72902a67d" (UID: "e2476a2b-8eb0-4b13-ae22-47c72902a67d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280026 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280070 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsw7w\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-kube-api-access-hsw7w\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280088 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280111 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280125 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280139 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280153 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2476a2b-8eb0-4b13-ae22-47c72902a67d-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.281613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.280167 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2476a2b-8eb0-4b13-ae22-47c72902a67d-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:23.458656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.458618 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 22:28:23.463298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:23.463271 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/simulator-w6-fb9597-kserve-958d57475-45f4h"] Nov 01 22:28:24.090468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.088917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:24.144467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.124088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" event={"ID":"e2476a2b-8eb0-4b13-ae22-47c72902a67d","Type":"ContainerDied","Data":"430dbdc11da43fce97afd70c6278d72f933fbe608dc89d448e63603bf4a17304"} Nov 01 22:28:24.144467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.124260 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq" Nov 01 22:28:24.173477 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:24.162659 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:24.226562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.226523 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:28:24.234070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.234034 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-f7ckq"] Nov 01 22:28:24.445011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.444487 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813df7b3-d181-4675-a695-af946f32a232" path="/var/lib/kubelet/pods/813df7b3-d181-4675-a695-af946f32a232/volumes" Nov 01 22:28:24.445202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.445013 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" path="/var/lib/kubelet/pods/e2476a2b-8eb0-4b13-ae22-47c72902a67d/volumes" Nov 01 22:28:24.536024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.535985 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:24.657316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:24.657277 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:25.188053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:25.188018 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:25.740468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:25.739588 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:25.953288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:25.953206 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:26.243462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:26.241301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:27.017468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:27.016311 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:27.356062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:27.355945 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:27.390164 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:27.389434 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:27.524468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:27.513803 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:28.045256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:28.045210 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:28.836414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:28.835618 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:29.021191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:29.021096 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:29.888694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:29.888652 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:30.743531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:30.743493 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:31.344518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.344482 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km"] Nov 01 22:28:31.345209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345190 5036 cpu_manager.go:414] "containerMap: removing container" podUID="055d8b49-77f8-4e2b-831b-3dd665a7d70e" containerName="revoke-keys" Nov 01 22:28:31.345209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345208 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055d8b49-77f8-4e2b-831b-3dd665a7d70e" containerName="revoke-keys" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345220 5036 cpu_manager.go:414] "containerMap: removing container" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345225 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345257 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345262 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345279 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7b7267a8-8175-48a0-8bd2-fdf2251d11e9" containerName="revoke-keys" Nov 01 22:28:31.345358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345285 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7267a8-8175-48a0-8bd2-fdf2251d11e9" containerName="revoke-keys" Nov 01 22:28:31.345736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345382 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7267a8-8175-48a0-8bd2-fdf2251d11e9" containerName="revoke-keys" Nov 01 22:28:31.345736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345396 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="055d8b49-77f8-4e2b-831b-3dd665a7d70e" containerName="revoke-keys" Nov 01 22:28:31.345736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345402 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="813df7b3-d181-4675-a695-af946f32a232" containerName="main" Nov 01 22:28:31.345736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.345410 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2476a2b-8eb0-4b13-ae22-47c72902a67d" containerName="istio-proxy" Nov 01 22:28:31.351023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.350999 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr"] Nov 01 22:28:31.351184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.351161 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.355740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.355719 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.361792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.361752 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km"] Nov 01 22:28:31.367651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.367627 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:28:31.368909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.368885 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:28:31.432529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.432489 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr"] Nov 01 22:28:31.456519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.456349 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h"] Nov 01 22:28:31.472367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.471759 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491517 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491586 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491639 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491739 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491853 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqxn\" (UniqueName: \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.493701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.491956 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72ht\" (UniqueName: \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.497174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.495771 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h"] Nov 01 22:28:31.593761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.593712 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72ht\" (UniqueName: \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.593964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.593860 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.594112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594089 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.594182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594160 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.594278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594250 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.594340 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594286 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd84p\" (UniqueName: \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.594414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594342 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.594749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqxn\" (UniqueName: \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.594749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.594740 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.595234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.595205 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.595487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.595353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.597773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.597699 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.597773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.597725 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.606978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.606951 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72ht\" (UniqueName: \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\") pod \"maas-api-revoke-sub-ca1b51ca-ht7km\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.607263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.607236 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqxn\" (UniqueName: \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\") pod \"maas-api-revoke-sub-617d2576-pjmfr\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.691503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.691420 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:31.696964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.696932 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.697211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.697194 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.697302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.697226 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd84p\" (UniqueName: \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.698050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.698023 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.700678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.700656 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.716393 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.716344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd84p\" (UniqueName: \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\") pod \"maas-api-revoke-sub-7e613288-frj9h\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:31.735505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.735479 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:31.772471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.762714 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:31.805302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:31.803703 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:32.056415 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:32.056378 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef19f521_ad16_4ce0_9779_36f19b3b28e9.slice/crio-e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09 WatchSource:0}: Error finding container e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09: Status 404 returned error can't find the container with id e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09 Nov 01 22:28:32.057014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.056992 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km"] Nov 01 22:28:32.125336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.125305 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr"] Nov 01 22:28:32.129436 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:32.129411 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc7db18_770c_45c8_b86d_775adceabe89.slice/crio-afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f WatchSource:0}: Error finding container afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f: Status 404 returned error can't find the container with id afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f Nov 01 22:28:32.171677 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:32.171652 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2162588_5a26_4be9_ba4d_56e90335d2d8.slice/crio-5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc WatchSource:0}: Error finding container 5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc: Status 404 returned error can't find the container with id 5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc Nov 01 22:28:32.172713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.172686 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h"] Nov 01 22:28:32.495136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.495100 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" event={"ID":"5cc7db18-770c-45c8-b86d-775adceabe89","Type":"ContainerStarted","Data":"afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f"} Nov 01 22:28:32.496999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.496974 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" event={"ID":"d2162588-5a26-4be9-ba4d-56e90335d2d8","Type":"ContainerStarted","Data":"5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc"} Nov 01 22:28:32.498904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.498881 5036 generic.go:356] "Generic (PLEG): container finished" podID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerID="613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5" exitCode=0 Nov 01 22:28:32.499024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.498963 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerDied","Data":"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5"} Nov 01 22:28:32.499716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.499699 5036 scope.go:122] "RemoveContainer" containerID="613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5" Nov 01 22:28:32.500912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.500884 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" event={"ID":"ef19f521-ad16-4ce0-9779-36f19b3b28e9","Type":"ContainerStarted","Data":"e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09"} Nov 01 22:28:32.502044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.502012 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:32.502140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.502103 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:32.502230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.502153 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:32.626715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:32.626680 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:33.305879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.305848 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:33.515220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.515181 5036 generic.go:356] "Generic (PLEG): container finished" podID="5cc7db18-770c-45c8-b86d-775adceabe89" containerID="ddbd481f70f7d5240a4bf94fb8a8a7ce2407c1eb862a475453de3fba9ae63ea9" exitCode=0 Nov 01 22:28:33.515616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.515262 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" event={"ID":"5cc7db18-770c-45c8-b86d-775adceabe89","Type":"ContainerDied","Data":"ddbd481f70f7d5240a4bf94fb8a8a7ce2407c1eb862a475453de3fba9ae63ea9"} Nov 01 22:28:33.517256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.517231 5036 generic.go:356] "Generic (PLEG): container finished" podID="d2162588-5a26-4be9-ba4d-56e90335d2d8" containerID="d34bbe3e1f69dc865730957708a968c4a7abda2261d810d473bffedc8d4b8b2d" exitCode=0 Nov 01 22:28:33.517391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.517308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" event={"ID":"d2162588-5a26-4be9-ba4d-56e90335d2d8","Type":"ContainerDied","Data":"d34bbe3e1f69dc865730957708a968c4a7abda2261d810d473bffedc8d4b8b2d"} Nov 01 22:28:33.520042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.520021 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerStarted","Data":"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5"} Nov 01 22:28:33.522064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.522044 5036 generic.go:356] "Generic (PLEG): container finished" podID="ef19f521-ad16-4ce0-9779-36f19b3b28e9" containerID="cb42a2958bc97ca95eea83df6bf94dadeb7b4806d7faf6161fba611ebb8f74ea" exitCode=0 Nov 01 22:28:33.522186 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.522077 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" event={"ID":"ef19f521-ad16-4ce0-9779-36f19b3b28e9","Type":"ContainerDied","Data":"cb42a2958bc97ca95eea83df6bf94dadeb7b4806d7faf6161fba611ebb8f74ea"} Nov 01 22:28:33.733531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.733494 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:28:33.734940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.734915 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:33.735024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.734968 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:33.953991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:33.953949 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:34.741479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:34.739409 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:34.741479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:34.739498 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:34.793543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:34.793260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:34.940482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:34.931693 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:34.940482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:34.934574 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:35.032914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.032762 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\") pod \"5cc7db18-770c-45c8-b86d-775adceabe89\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " Nov 01 22:28:35.033426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.033402 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\" (UniqueName: \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\") pod \"5cc7db18-770c-45c8-b86d-775adceabe89\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " Nov 01 22:28:35.033840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.033817 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\") pod \"5cc7db18-770c-45c8-b86d-775adceabe89\" (UID: \"5cc7db18-770c-45c8-b86d-775adceabe89\") " Nov 01 22:28:35.034295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.034266 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca" pod "5cc7db18-770c-45c8-b86d-775adceabe89" (UID: "5cc7db18-770c-45c8-b86d-775adceabe89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:35.034403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.034343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\") pod \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " Nov 01 22:28:35.034812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.034788 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\" (UniqueName: \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\") pod \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " Nov 01 22:28:35.035202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.035177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\") pod \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\" (UID: \"ef19f521-ad16-4ce0-9779-36f19b3b28e9\") " Nov 01 22:28:35.035640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.035614 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca" pod "ef19f521-ad16-4ce0-9779-36f19b3b28e9" (UID: "ef19f521-ad16-4ce0-9779-36f19b3b28e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:35.035744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.035701 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca" pod "5cc7db18-770c-45c8-b86d-775adceabe89" (UID: "5cc7db18-770c-45c8-b86d-775adceabe89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:35.036688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.036661 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn" pod "5cc7db18-770c-45c8-b86d-775adceabe89" (UID: "5cc7db18-770c-45c8-b86d-775adceabe89"). InnerVolumeSpecName "kube-api-access-8vqxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:35.036912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.036891 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca" pod "ef19f521-ad16-4ce0-9779-36f19b3b28e9" (UID: "ef19f521-ad16-4ce0-9779-36f19b3b28e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:35.037317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.037296 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqxn\" (UniqueName: \"kubernetes.io/projected/5cc7db18-770c-45c8-b86d-775adceabe89-kube-api-access-8vqxn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.037394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.037323 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.037394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.037337 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.037394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.037351 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19f521-ad16-4ce0-9779-36f19b3b28e9-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.037394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.037365 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc7db18-770c-45c8-b86d-775adceabe89-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.039057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.039037 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht" pod "ef19f521-ad16-4ce0-9779-36f19b3b28e9" (UID: "ef19f521-ad16-4ce0-9779-36f19b3b28e9"). InnerVolumeSpecName "kube-api-access-v72ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:35.042560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.042539 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:35.138270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.138175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\") pod \"d2162588-5a26-4be9-ba4d-56e90335d2d8\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " Nov 01 22:28:35.138436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.138337 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\") pod \"d2162588-5a26-4be9-ba4d-56e90335d2d8\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " Nov 01 22:28:35.138436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.138398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\" (UniqueName: \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\") pod \"d2162588-5a26-4be9-ba4d-56e90335d2d8\" (UID: \"d2162588-5a26-4be9-ba4d-56e90335d2d8\") " Nov 01 22:28:35.139193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.139164 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72ht\" (UniqueName: \"kubernetes.io/projected/ef19f521-ad16-4ce0-9779-36f19b3b28e9-kube-api-access-v72ht\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.139193 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.139172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca" pod "d2162588-5a26-4be9-ba4d-56e90335d2d8" (UID: "d2162588-5a26-4be9-ba4d-56e90335d2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:35.140832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.140811 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca" pod "d2162588-5a26-4be9-ba4d-56e90335d2d8" (UID: "d2162588-5a26-4be9-ba4d-56e90335d2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:35.141035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.141013 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p" pod "d2162588-5a26-4be9-ba4d-56e90335d2d8" (UID: "d2162588-5a26-4be9-ba4d-56e90335d2d8"). InnerVolumeSpecName "kube-api-access-pd84p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:35.240873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.240839 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.240873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.240869 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2162588-5a26-4be9-ba4d-56e90335d2d8-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.240873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.240881 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd84p\" (UniqueName: \"kubernetes.io/projected/d2162588-5a26-4be9-ba4d-56e90335d2d8-kube-api-access-pd84p\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.542894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.540662 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:35.633528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.627561 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" event={"ID":"ef19f521-ad16-4ce0-9779-36f19b3b28e9","Type":"ContainerDied","Data":"e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09"} Nov 01 22:28:35.633528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.627632 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e313857976fabff94fca3b8ad9c2fa4bbcaebef7f8368653a54fef88711d7d09" Nov 01 22:28:35.633528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.627748 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km" Nov 01 22:28:35.635626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.635600 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" event={"ID":"5cc7db18-770c-45c8-b86d-775adceabe89","Type":"ContainerDied","Data":"afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f"} Nov 01 22:28:35.636028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.636007 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdcb00d0199f971aae5f6e480d33e6af8ab78cbd2991bda4820784232f0725f" Nov 01 22:28:35.636501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.636170 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr" Nov 01 22:28:35.639006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.638987 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" event={"ID":"d2162588-5a26-4be9-ba4d-56e90335d2d8","Type":"ContainerDied","Data":"5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc"} Nov 01 22:28:35.639148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.639135 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fac86f0d20bb210967f8bfda777898dfbaba78c5c93b70fe56c5382de06b6dc" Nov 01 22:28:35.639252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.639243 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h" Nov 01 22:28:35.733844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.733810 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:35.734028 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:35.733866 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:36.026033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.025983 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn"] Nov 01 22:28:36.030632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.030597 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b609be6-dwddn"] Nov 01 22:28:36.172957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.172926 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:36.453041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.452141 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d40012f-4004-438d-bec0-020f6544dc96" path="/var/lib/kubelet/pods/3d40012f-4004-438d-bec0-020f6544dc96/volumes" Nov 01 22:28:36.571909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.571885 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 22:28:36.655505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.655425 5036 generic.go:356] "Generic (PLEG): container finished" podID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerID="1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c" exitCode=0 Nov 01 22:28:36.655505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.655493 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" Nov 01 22:28:36.655789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.655518 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" event={"ID":"dbfc35e8-62a7-46e5-9d9d-dedd613e0444","Type":"ContainerDied","Data":"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c"} Nov 01 22:28:36.655789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.655598 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h" event={"ID":"dbfc35e8-62a7-46e5-9d9d-dedd613e0444","Type":"ContainerDied","Data":"36963e1d4cb58e636a96aa6c523e5ccfaffdce02e06ea0442f5a3907c3f59632"} Nov 01 22:28:36.655789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.655630 5036 scope.go:122] "RemoveContainer" containerID="1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c" Nov 01 22:28:36.667423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.667397 5036 scope.go:122] "RemoveContainer" containerID="1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c" Nov 01 22:28:36.667711 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:36.667686 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c\": container with ID starting with 1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c not found: ID does not exist" containerID="1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c" Nov 01 22:28:36.667809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.667717 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c"} err="failed to get container status \"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c\": rpc error: code = NotFound desc = could not find container \"1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c\": container with ID starting with 1373ec6f77378956109d8f5cc40e97509eab9e82866cfda0cae2538ce4e6133c not found: ID does not exist" Nov 01 22:28:36.676334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.676303 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\") pod \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " Nov 01 22:28:36.691493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.676368 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\" (UniqueName: \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\") pod \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\" (UID: \"dbfc35e8-62a7-46e5-9d9d-dedd613e0444\") " Nov 01 22:28:36.691493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.676915 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume" pod "dbfc35e8-62a7-46e5-9d9d-dedd613e0444" (UID: "dbfc35e8-62a7-46e5-9d9d-dedd613e0444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:36.691493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.677162 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.718919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.718875 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff" pod "dbfc35e8-62a7-46e5-9d9d-dedd613e0444" (UID: "dbfc35e8-62a7-46e5-9d9d-dedd613e0444"). InnerVolumeSpecName "kube-api-access-ls7ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:36.733498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.733465 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:36.733621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.733520 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:36.795469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.790773 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls7ff\" (UniqueName: \"kubernetes.io/projected/dbfc35e8-62a7-46e5-9d9d-dedd613e0444-kube-api-access-ls7ff\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.929955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:36.929866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:37.042614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.042580 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 22:28:37.050464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.049195 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb9597-b5958d4dc-2t46h"] Nov 01 22:28:37.729999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.701421 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:28:37.745468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.742948 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:37.745468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.743029 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:37.820469 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:37.811861 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:37.891343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:37.866162 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:38.049511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.049453 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:28:38.050227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050209 5036 cpu_manager.go:414] "containerMap: removing container" podUID="5cc7db18-770c-45c8-b86d-775adceabe89" containerName="revoke-keys" Nov 01 22:28:38.050277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050229 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc7db18-770c-45c8-b86d-775adceabe89" containerName="revoke-keys" Nov 01 22:28:38.050277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050265 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" Nov 01 22:28:38.050277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050271 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" Nov 01 22:28:38.050377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050289 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ef19f521-ad16-4ce0-9779-36f19b3b28e9" containerName="revoke-keys" Nov 01 22:28:38.050377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050295 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef19f521-ad16-4ce0-9779-36f19b3b28e9" containerName="revoke-keys" Nov 01 22:28:38.050377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050309 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d2162588-5a26-4be9-ba4d-56e90335d2d8" containerName="revoke-keys" Nov 01 22:28:38.050377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050316 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2162588-5a26-4be9-ba4d-56e90335d2d8" containerName="revoke-keys" Nov 01 22:28:38.050532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050429 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc7db18-770c-45c8-b86d-775adceabe89" containerName="revoke-keys" Nov 01 22:28:38.050532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050457 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" containerName="payload-pre-processing" Nov 01 22:28:38.050532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050470 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef19f521-ad16-4ce0-9779-36f19b3b28e9" containerName="revoke-keys" Nov 01 22:28:38.050532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.050482 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2162588-5a26-4be9-ba4d-56e90335d2d8" containerName="revoke-keys" Nov 01 22:28:38.055827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.055804 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.060991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.060970 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb9597-openshift-default-dockercfg-lhqww\"" Nov 01 22:28:38.067927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.067874 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:28:38.123481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.122646 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" containerID="cri-o://e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b" gracePeriod=45 Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198465 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198552 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198735 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwj8\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198856 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.198892 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.199010 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.199232 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.208534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.199292 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302523 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302581 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302695 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302752 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302796 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302824 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302881 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.302922 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.303059 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwj8\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.304189 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.304511 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.304756 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.309515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.304986 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.318905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.317655 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.320211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.319117 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" containerID="cri-o://5f329d9ba1bc4c50ec62ae6ba4f4c4e499cd4fd6f9ee1799634df4416290d13c" gracePeriod=45 Nov 01 22:28:38.321336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.321220 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.324300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.323966 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.327241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.327200 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwj8\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.328047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.327984 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.444486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.442344 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:38.463283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.463237 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfc35e8-62a7-46e5-9d9d-dedd613e0444" path="/var/lib/kubelet/pods/dbfc35e8-62a7-46e5-9d9d-dedd613e0444/volumes" Nov 01 22:28:38.511499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.476403 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Nov 01 22:28:38.542730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.542419 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:28:38.639158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639114 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.639339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639320 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.639423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639396 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\" (UniqueName: \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.639559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639540 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.639632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.639699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.639684 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\") pod \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\" (UID: \"1edb3f3e-2731-4d2e-ab4e-4c5c83703602\") " Nov 01 22:28:38.640864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.640542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:38.640864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.640688 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:38.640864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.640783 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:38.640864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.640810 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.643553 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.643531 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "kube-api-access-92824". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:38.643883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.643843 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:38.643883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.643854 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs" pod "1edb3f3e-2731-4d2e-ab4e-4c5c83703602" (UID: "1edb3f3e-2731-4d2e-ab4e-4c5c83703602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:38.729286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.729249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:38.733037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.733008 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 01 22:28:38.733255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.733227 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 01 22:28:38.781570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.760126 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.781570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.760165 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92824\" (UniqueName: \"kubernetes.io/projected/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-kube-api-access-92824\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.781570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.760184 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.781570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.760199 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.781570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.760214 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edb3f3e-2731-4d2e-ab4e-4c5c83703602-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:38.873787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.873749 5036 generic.go:356] "Generic (PLEG): container finished" podID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerID="e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b" exitCode=0 Nov 01 22:28:38.874224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.874203 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" event={"ID":"1edb3f3e-2731-4d2e-ab4e-4c5c83703602","Type":"ContainerDied","Data":"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b"} Nov 01 22:28:38.874388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.874372 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" event={"ID":"1edb3f3e-2731-4d2e-ab4e-4c5c83703602","Type":"ContainerDied","Data":"5623d6b44211a2e3e5adaf629959367418256e5345086c7e4b4e70580473e1e0"} Nov 01 22:28:38.874531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.874517 5036 scope.go:122] "RemoveContainer" containerID="e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b" Nov 01 22:28:38.875872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.875845 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2" Nov 01 22:28:38.948003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.947973 5036 generic.go:356] "Generic (PLEG): container finished" podID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerID="5f329d9ba1bc4c50ec62ae6ba4f4c4e499cd4fd6f9ee1799634df4416290d13c" exitCode=0 Nov 01 22:28:38.948402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.948379 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" event={"ID":"eb679209-52cd-41f0-bde3-9d6a6549d4ad","Type":"ContainerDied","Data":"5f329d9ba1bc4c50ec62ae6ba4f4c4e499cd4fd6f9ee1799634df4416290d13c"} Nov 01 22:28:38.961091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.961057 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 22:28:38.969887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.969864 5036 scope.go:122] "RemoveContainer" containerID="e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b" Nov 01 22:28:38.970067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.969952 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-w2-f033b1-kserve-f7774fbf8-qgzz2"] Nov 01 22:28:38.970560 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:38.970227 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b\": container with ID starting with e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b not found: ID does not exist" containerID="e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b" Nov 01 22:28:38.970560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.970264 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b"} err="failed to get container status \"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b\": rpc error: code = NotFound desc = could not find container \"e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b\": container with ID starting with e76692880767b9367c95b041a496d83426a6bc64076d39292d489b90a5d7e33b not found: ID does not exist" Nov 01 22:28:38.974590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:38.974571 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:28:39.041289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.041260 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:28:39.046639 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:28:39.046591 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c55a90_89bb_45c4_b44a_3e9400d86c46.slice/crio-387266dc4dd21cdc6b07eab729c9bf1c1ae190ce8e8ea9bb97d4ccf099f0d4ac WatchSource:0}: Error finding container 387266dc4dd21cdc6b07eab729c9bf1c1ae190ce8e8ea9bb97d4ccf099f0d4ac: Status 404 returned error can't find the container with id 387266dc4dd21cdc6b07eab729c9bf1c1ae190ce8e8ea9bb97d4ccf099f0d4ac Nov 01 22:28:39.049868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.049808 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:39.050374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.050346 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:39.050555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.050532 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:28:39.149414 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:39.149382 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:39.176376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176352 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.176540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176513 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.176620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176552 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.176620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.176751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176682 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.176751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.176715 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\" (UniqueName: \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\") pod \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\" (UID: \"eb679209-52cd-41f0-bde3-9d6a6549d4ad\") " Nov 01 22:28:39.200687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.194538 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:39.200687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.194790 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:39.200915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.200886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:39.215489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.213324 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "kube-api-access-dj74h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:39.215489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.213568 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:39.215716 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.215650 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm" pod "eb679209-52cd-41f0-bde3-9d6a6549d4ad" (UID: "eb679209-52cd-41f0-bde3-9d6a6549d4ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:39.285476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.280117 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.285747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.285724 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.285950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.285936 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.286055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.286043 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.286157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.286141 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj74h\" (UniqueName: \"kubernetes.io/projected/eb679209-52cd-41f0-bde3-9d6a6549d4ad-kube-api-access-dj74h\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.286257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.286244 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb679209-52cd-41f0-bde3-9d6a6549d4ad-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:39.389734 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:39.389696 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:39.415976 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:39.415924 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:39.541997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.541818 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:39.663212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.663175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:39.852985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:39.852896 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:40.006342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.006285 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" event={"ID":"24c55a90-89bb-45c4-b44a-3e9400d86c46","Type":"ContainerStarted","Data":"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099"} Nov 01 22:28:40.006651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.006615 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" event={"ID":"24c55a90-89bb-45c4-b44a-3e9400d86c46","Type":"ContainerStarted","Data":"387266dc4dd21cdc6b07eab729c9bf1c1ae190ce8e8ea9bb97d4ccf099f0d4ac"} Nov 01 22:28:40.018360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.018103 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" event={"ID":"eb679209-52cd-41f0-bde3-9d6a6549d4ad","Type":"ContainerDied","Data":"013e749f32f4f11381203f1de42891fb6c931c70c45dc8145fe51c2b73a2d201"} Nov 01 22:28:40.018360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.018158 5036 scope.go:122] "RemoveContainer" containerID="5f329d9ba1bc4c50ec62ae6ba4f4c4e499cd4fd6f9ee1799634df4416290d13c" Nov 01 22:28:40.018360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.018296 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg" Nov 01 22:28:40.041175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.041114 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podStartSLOduration=2.041093205 podStartE2EDuration="2.041093205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:40.036233427 +0000 UTC m=+2673.729213927" watchObservedRunningTime="2026-11-01 22:28:40.041093205 +0000 UTC m=+2673.734073693" Nov 01 22:28:40.086832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.086795 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 22:28:40.095109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.095067 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/premium-w2-f033b1-kserve-7d4b8d7f6c-8bxhg"] Nov 01 22:28:40.097521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.097008 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:28:40.101195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.101170 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:28:40.424026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.423992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:40.446149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.446112 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" path="/var/lib/kubelet/pods/1edb3f3e-2731-4d2e-ab4e-4c5c83703602/volumes" Nov 01 22:28:40.447491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.446847 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" path="/var/lib/kubelet/pods/eb679209-52cd-41f0-bde3-9d6a6549d4ad/volumes" Nov 01 22:28:40.447491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.447320 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:40.447673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.447543 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:40.447673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.447610 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:40.600467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:40.592876 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:41.150673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.150380 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:41.207344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.207300 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:28:41.245472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.243710 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerName="maas-api" containerID="cri-o://c5127b64be6f81374e37938c864c4ec3b894cf1a0446330b91f03f7dc0dc502d" gracePeriod=30 Nov 01 22:28:41.563530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.531354 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:41.563530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.531407 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:41.608039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.595757 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:41.648094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.648047 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.78:8443/health\": dial tcp 10.132.1.78:8443: connect: connection refused" Nov 01 22:28:41.771482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.769931 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:41.919195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.918485 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f"] Nov 01 22:28:41.937543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:41.927037 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a00c6a-m2b7f"] Nov 01 22:28:42.056295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:42.056055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:42.493865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:42.493826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:42.573496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:42.572068 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:42.573496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:42.572133 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:43.216721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.216676 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:43.319351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.318070 5036 generic.go:356] "Generic (PLEG): container finished" podID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerID="c5127b64be6f81374e37938c864c4ec3b894cf1a0446330b91f03f7dc0dc502d" exitCode=0 Nov 01 22:28:43.447919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.427017 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7267a8-8175-48a0-8bd2-fdf2251d11e9" path="/var/lib/kubelet/pods/7b7267a8-8175-48a0-8bd2-fdf2251d11e9/volumes" Nov 01 22:28:43.447919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.427564 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" event={"ID":"c89e6f84-ce0b-40cf-a029-e318dc428ee9","Type":"ContainerDied","Data":"c5127b64be6f81374e37938c864c4ec3b894cf1a0446330b91f03f7dc0dc502d"} Nov 01 22:28:43.510767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.510735 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:43.514466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.511320 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:43.523818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.523782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:43.713083 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.713055 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:28:43.714670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.714642 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:43.767590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.767505 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\") pod \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " Nov 01 22:28:43.769513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.769484 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\") pod \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " Nov 01 22:28:43.769762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.769740 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\" (UniqueName: \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\") pod \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " Nov 01 22:28:43.770336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.770017 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca" pod "c89e6f84-ce0b-40cf-a029-e318dc428ee9" (UID: "c89e6f84-ce0b-40cf-a029-e318dc428ee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:43.770464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.770106 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls" pod "c89e6f84-ce0b-40cf-a029-e318dc428ee9" (UID: "c89e6f84-ce0b-40cf-a029-e318dc428ee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:43.770635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.770608 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\") pod \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\" (UID: \"c89e6f84-ce0b-40cf-a029-e318dc428ee9\") " Nov 01 22:28:43.771380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.771353 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:43.771511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.771384 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89e6f84-ce0b-40cf-a029-e318dc428ee9-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:43.772773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.772743 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs" pod "c89e6f84-ce0b-40cf-a029-e318dc428ee9" (UID: "c89e6f84-ce0b-40cf-a029-e318dc428ee9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:43.772947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.772904 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj" pod "c89e6f84-ce0b-40cf-a029-e318dc428ee9" (UID: "c89e6f84-ce0b-40cf-a029-e318dc428ee9"). InnerVolumeSpecName "kube-api-access-b9srj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:43.856354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.856301 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:43.873574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.873536 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9srj\" (UniqueName: \"kubernetes.io/projected/c89e6f84-ce0b-40cf-a029-e318dc428ee9-kube-api-access-b9srj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:43.873574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.873575 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89e6f84-ce0b-40cf-a029-e318dc428ee9-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:43.999522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:43.999326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:44.335346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.335303 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" event={"ID":"c89e6f84-ce0b-40cf-a029-e318dc428ee9","Type":"ContainerDied","Data":"490db5440b9b3dc83f0efbeba24a9c902fc3c294e3ad92bc20c540525ae1edc8"} Nov 01 22:28:44.335576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.335382 5036 scope.go:122] "RemoveContainer" containerID="c5127b64be6f81374e37938c864c4ec3b894cf1a0446330b91f03f7dc0dc502d" Nov 01 22:28:44.335675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.335579 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg" Nov 01 22:28:44.364947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.364919 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:28:44.370626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.370603 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a00c6a-76b496489d-knfxg"] Nov 01 22:28:44.440733 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.440702 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" path="/var/lib/kubelet/pods/c89e6f84-ce0b-40cf-a029-e318dc428ee9/volumes" Nov 01 22:28:44.447187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.447159 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:44.447274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.447230 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:44.613120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.612831 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:44.763946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:44.763909 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:45.446379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:45.446337 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:45.446609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:45.446408 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:45.557826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:45.557789 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:46.260490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:46.259284 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:46.446485 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:46.446436 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:46.446672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:46.446515 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:47.040693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.040661 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m"] Nov 01 22:28:47.046278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.046251 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ff4f5-wdm2m"] Nov 01 22:28:47.141314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.141260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:47.273096 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:47.273063 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:47.446724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.446636 5036 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 01 22:28:47.446996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.446957 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 01 22:28:47.824898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:47.824854 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:47.928820 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:47.928778 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:48.148003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.147910 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:48.443616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.443515 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac11d2b2-4a0f-4849-989a-6b737d0bc156" path="/var/lib/kubelet/pods/ac11d2b2-4a0f-4849-989a-6b737d0bc156/volumes" Nov 01 22:28:48.554303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.554263 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:48.788783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.788747 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:48.789669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.789600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:48.790723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:48.790661 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:28:48.866540 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:48.866421 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:28:49.365478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:49.365403 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:50.026958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:50.026918 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:50.663458 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:50.663413 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:51.164795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:51.164757 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:51.433266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:51.433183 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:52.222878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:52.222837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:52.859800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:52.859766 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:53.013823 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.013788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:53.039160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.039129 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t"] Nov 01 22:28:53.043726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.043701 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb9597-xhq8t"] Nov 01 22:28:53.159520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.157957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:53.437970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.437826 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:53.588305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.588272 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6"] Nov 01 22:28:53.596867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.596829 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:53.603685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.597325 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f033b1-x4nc6"] Nov 01 22:28:53.755573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:53.755403 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:54.106686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:54.106650 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:54.362656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:54.362566 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:54.446419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:54.446372 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055d8b49-77f8-4e2b-831b-3dd665a7d70e" path="/var/lib/kubelet/pods/055d8b49-77f8-4e2b-831b-3dd665a7d70e/volumes" Nov 01 22:28:54.447199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:54.447171 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e84511-dd47-4e6c-8866-5cfbebf1d415" path="/var/lib/kubelet/pods/40e84511-dd47-4e6c-8866-5cfbebf1d415/volumes" Nov 01 22:28:54.499200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:54.499166 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:55.096594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:55.096543 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:55.828305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:55.828271 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:56.631091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:56.631055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:57.320468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:57.319970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:58.065629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.065566 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4"] Nov 01 22:28:58.093540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.093363 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518b56ba-nk4z4"] Nov 01 22:28:58.132095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.132062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:58.207655 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:28:58.207363 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:28:58.405903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.405859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:58.445631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.445592 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed428a9-fcb6-4766-9547-030c1935c347" path="/var/lib/kubelet/pods/fed428a9-fcb6-4766-9547-030c1935c347/volumes" Nov 01 22:28:58.603381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:58.603337 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:59.099291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:59.099254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:59.826719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:28:59.826681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:00.038249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:00.038213 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:00.189181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:00.189088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:00.955634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:00.955582 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:01.087770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:01.087731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:01.216410 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:01.216317 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:01.831998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:01.831967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:01.970350 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:01.970307 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:02.092676 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:02.083917 5036 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e84511_dd47_4e6c_8866_5cfbebf1d415.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:29:03.892645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:03.892609 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:04.146031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:04.145923 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:04.938706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:04.938673 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:05.810095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:05.810057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:06.528742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:06.528701 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:07.231191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:07.231145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:07.916309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:07.916275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:07.922388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:07.922367 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:29:08.027274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:08.027231 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l"] Nov 01 22:29:08.036007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:08.035976 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ca4a5e-jpw6l"] Nov 01 22:29:08.441433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:08.441399 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81975588-5e48-4d1b-b9b7-940bd0dfcee1" path="/var/lib/kubelet/pods/81975588-5e48-4d1b-b9b7-940bd0dfcee1/volumes" Nov 01 22:29:08.679341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:08.679313 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:29:08.705771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:08.705675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:09.740597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:09.740557 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:10.453189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:10.453154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:11.143685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.143647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:11.644339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.644307 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8"] Nov 01 22:29:11.645173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645148 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" Nov 01 22:29:11.645173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645172 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" Nov 01 22:29:11.645370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645208 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" Nov 01 22:29:11.645370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645217 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" Nov 01 22:29:11.645370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645255 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerName="maas-api" Nov 01 22:29:11.645370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645263 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerName="maas-api" Nov 01 22:29:11.645619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645455 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89e6f84-ce0b-40cf-a029-e318dc428ee9" containerName="maas-api" Nov 01 22:29:11.645619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645483 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb679209-52cd-41f0-bde3-9d6a6549d4ad" containerName="main" Nov 01 22:29:11.645619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.645496 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb3f3e-2731-4d2e-ab4e-4c5c83703602" containerName="main" Nov 01 22:29:11.651306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.651277 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.656046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.656015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8"] Nov 01 22:29:11.692080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.692055 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:29:11.692222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.692122 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:29:11.788388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.788357 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.788592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.788514 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vd7\" (UniqueName: \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.788592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.788543 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.889850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.889811 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.890032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.889952 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vd7\" (UniqueName: \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.890032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.889982 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.890731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.890696 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.892413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.892378 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.902231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.902167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vd7\" (UniqueName: \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:11.973387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:11.973354 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:12.168307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:12.141414 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:12.231104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:12.231059 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:29:12.268858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:12.268827 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8"] Nov 01 22:29:12.269235 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:29:12.269214 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04412c40_7832_4cdb_b2a7_6447fc47537e.slice/crio-7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611 WatchSource:0}: Error finding container 7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611: Status 404 returned error can't find the container with id 7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611 Nov 01 22:29:13.154476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:13.150789 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:13.207472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:13.206937 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" event={"ID":"04412c40-7832-4cdb-b2a7-6447fc47537e","Type":"ContainerStarted","Data":"24ef50d7fedeb4e13cb100814bc9782b0afe8b292cfeb4652d8b97d0b16dfba7"} Nov 01 22:29:13.207472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:13.207005 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" event={"ID":"04412c40-7832-4cdb-b2a7-6447fc47537e","Type":"ContainerStarted","Data":"7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611"} Nov 01 22:29:13.261466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:13.259940 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" podStartSLOduration=2.259917861 podStartE2EDuration="2.259917861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:13.252639153 +0000 UTC m=+2706.945619640" watchObservedRunningTime="2026-11-01 22:29:13.259917861 +0000 UTC m=+2706.952898348" Nov 01 22:29:13.913702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:13.913664 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:14.051816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:14.051775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:14.315979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:14.312175 5036 generic.go:356] "Generic (PLEG): container finished" podID="04412c40-7832-4cdb-b2a7-6447fc47537e" containerID="24ef50d7fedeb4e13cb100814bc9782b0afe8b292cfeb4652d8b97d0b16dfba7" exitCode=0 Nov 01 22:29:14.315979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:14.312549 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" event={"ID":"04412c40-7832-4cdb-b2a7-6447fc47537e","Type":"ContainerDied","Data":"24ef50d7fedeb4e13cb100814bc9782b0afe8b292cfeb4652d8b97d0b16dfba7"} Nov 01 22:29:14.716868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:14.716768 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.547034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.547009 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:15.606338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.606248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.641618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.623918 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\" (UniqueName: \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\") pod \"04412c40-7832-4cdb-b2a7-6447fc47537e\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " Nov 01 22:29:15.664002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.656685 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\") pod \"04412c40-7832-4cdb-b2a7-6447fc47537e\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " Nov 01 22:29:15.664002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.656755 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\") pod \"04412c40-7832-4cdb-b2a7-6447fc47537e\" (UID: \"04412c40-7832-4cdb-b2a7-6447fc47537e\") " Nov 01 22:29:15.665650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.664588 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca" pod "04412c40-7832-4cdb-b2a7-6447fc47537e" (UID: "04412c40-7832-4cdb-b2a7-6447fc47537e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:15.670762 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.670718 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7" pod "04412c40-7832-4cdb-b2a7-6447fc47537e" (UID: "04412c40-7832-4cdb-b2a7-6447fc47537e"). InnerVolumeSpecName "kube-api-access-w5vd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:15.674661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.674629 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca" pod "04412c40-7832-4cdb-b2a7-6447fc47537e" (UID: "04412c40-7832-4cdb-b2a7-6447fc47537e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:15.759316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.759280 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:15.759316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.759314 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04412c40-7832-4cdb-b2a7-6447fc47537e-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:15.759316 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:15.759325 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5vd7\" (UniqueName: \"kubernetes.io/projected/04412c40-7832-4cdb-b2a7-6447fc47537e-kube-api-access-w5vd7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:16.333425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:16.333391 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" Nov 01 22:29:16.333644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:16.333423 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8" event={"ID":"04412c40-7832-4cdb-b2a7-6447fc47537e","Type":"ContainerDied","Data":"7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611"} Nov 01 22:29:16.333644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:16.333514 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dc6b85ddca89b418548bae03cfbce6414e54063d37809e3f0026ec3d572c611" Nov 01 22:29:16.582763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:16.582726 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:17.644651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:17.644608 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:18.054799 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.054760 5036 scope.go:122] "RemoveContainer" containerID="774cf26014c657e6a89a22a41123467c7ae090722e0983af848420b04b52b3df" Nov 01 22:29:18.072077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.072041 5036 scope.go:122] "RemoveContainer" containerID="fa52deab85a4dda2b3b1107b9a3cb945720f885c03c35894483cba32709cb93b" Nov 01 22:29:18.109681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.109657 5036 scope.go:122] "RemoveContainer" containerID="a217ffffa71842b02c1d02dc84964bcb0edbb72e7cc2df75295a0bff6d7c0d69" Nov 01 22:29:18.120723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.120687 5036 scope.go:122] "RemoveContainer" containerID="46ab25c3f09d9a4cb6b8286d0b8ec8408dd9585e7338551e0015438afe60dc97" Nov 01 22:29:18.426589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.426495 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:18.557884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.557845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:18.666653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.666613 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 22:29:18.667413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.667375 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" containerID="cri-o://e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8" gracePeriod=30 Nov 01 22:29:18.731959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.731834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:18.880134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:18.880043 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:19.035822 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:19.035749 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:29:19.310040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.309940 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:19.437387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.437350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:19.577526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.572613 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:29:19.577526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.573559 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" containerID="cri-o://3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099" gracePeriod=30 Nov 01 22:29:19.621625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.609866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:19.794558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:19.781998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:20.244050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:20.244015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:20.371373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:20.371332 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:20.984106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:20.984010 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:21.656579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:21.656539 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:22.352866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:22.352817 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:22.441483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:22.440923 5036 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:29:22.441483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:22.441004 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:29:23.012135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.012092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:23.143893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.143858 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:23.906019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.905991 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.921890 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.921951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922010 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922138 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922219 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922433 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.923612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.922643 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\") pod \"b4c5dadb-6e55-4c39-927a-3742c70b1584\" (UID: \"b4c5dadb-6e55-4c39-927a-3742c70b1584\") " Nov 01 22:29:23.924700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.924301 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.925247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.924988 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:23.925693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.925427 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.925693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.925485 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.925864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.925727 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.927652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.927579 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:29:23.928232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.928201 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "kube-api-access-x8jsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:23.929098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.929057 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:23.929302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:23.929269 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy" pod "b4c5dadb-6e55-4c39-927a-3742c70b1584" (UID: "b4c5dadb-6e55-4c39-927a-3742c70b1584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.007552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.007517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030794 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jsv\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-kube-api-access-x8jsv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030830 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030845 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030857 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030868 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c5dadb-6e55-4c39-927a-3742c70b1584-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030879 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030893 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030904 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c5dadb-6e55-4c39-927a-3742c70b1584-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.038465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.030917 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c5dadb-6e55-4c39-927a-3742c70b1584-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.465128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.465095 5036 generic.go:356] "Generic (PLEG): container finished" podID="4f28f366-969c-4428-b4d8-9956897cc5da" containerID="71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2" exitCode=0 Nov 01 22:29:24.465356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.465143 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerDied","Data":"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2"} Nov 01 22:29:24.465843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.465823 5036 scope.go:122] "RemoveContainer" containerID="71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2" Nov 01 22:29:24.466900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.466878 5036 generic.go:356] "Generic (PLEG): container finished" podID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerID="e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8" exitCode=0 Nov 01 22:29:24.467004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.466933 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" event={"ID":"b4c5dadb-6e55-4c39-927a-3742c70b1584","Type":"ContainerDied","Data":"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8"} Nov 01 22:29:24.467004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.466973 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" event={"ID":"b4c5dadb-6e55-4c39-927a-3742c70b1584","Type":"ContainerDied","Data":"bc3c426b90044e818f2fd2de49740d52fe4137ed5f15405d649dcf0631fe81a7"} Nov 01 22:29:24.467004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.466990 5036 scope.go:122] "RemoveContainer" containerID="e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8" Nov 01 22:29:24.467151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.466976 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm" Nov 01 22:29:24.467899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.467871 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:29:24.468003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.467952 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:29:24.468003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.467987 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:29:24.479462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.479423 5036 scope.go:122] "RemoveContainer" containerID="e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8" Nov 01 22:29:24.479818 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:24.479796 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8\": container with ID starting with e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8 not found: ID does not exist" containerID="e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8" Nov 01 22:29:24.479902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.479832 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8"} err="failed to get container status \"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8\": rpc error: code = NotFound desc = could not find container \"e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8\": container with ID starting with e5fff97ac922fd40a358e09fa86d10f98adb0a9659465af22d459fd0febfebf8 not found: ID does not exist" Nov 01 22:29:24.522948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.515152 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 22:29:24.522948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.521818 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-pt5wm"] Nov 01 22:29:24.773830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:24.773788 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:25.153557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.153526 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:29:25.249029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.248997 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.249214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249054 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.249214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249124 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.249214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.249406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249234 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.249406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249279 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:25.249406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249330 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249457 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249465 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\") pod \"24c55a90-89bb-45c4-b44a-3e9400d86c46\" (UID: \"24c55a90-89bb-45c4-b44a-3e9400d86c46\") " Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249544 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249760 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:25.250006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.249934 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:25.250438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.250335 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.250438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.250363 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.250438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.250378 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.250438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.250393 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.250438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.250410 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c55a90-89bb-45c4-b44a-3e9400d86c46-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.251872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.251849 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "kube-api-access-dnwj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:25.251997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.251885 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:29:25.252044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.252005 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:25.252087 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.252055 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token" pod "24c55a90-89bb-45c4-b44a-3e9400d86c46" (UID: "24c55a90-89bb-45c4-b44a-3e9400d86c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:25.351948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.351916 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnwj8\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-kube-api-access-dnwj8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.351948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.351947 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.352164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.351965 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.352164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.351980 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c55a90-89bb-45c4-b44a-3e9400d86c46-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.477497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.477463 5036 generic.go:356] "Generic (PLEG): container finished" podID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerID="3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099" exitCode=0 Nov 01 22:29:25.477711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.477502 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" event={"ID":"24c55a90-89bb-45c4-b44a-3e9400d86c46","Type":"ContainerDied","Data":"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099"} Nov 01 22:29:25.477711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.477561 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" Nov 01 22:29:25.477711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.477569 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p" event={"ID":"24c55a90-89bb-45c4-b44a-3e9400d86c46","Type":"ContainerDied","Data":"387266dc4dd21cdc6b07eab729c9bf1c1ae190ce8e8ea9bb97d4ccf099f0d4ac"} Nov 01 22:29:25.477711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.477599 5036 scope.go:122] "RemoveContainer" containerID="3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099" Nov 01 22:29:25.481066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.481002 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerStarted","Data":"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9"} Nov 01 22:29:25.492967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.492949 5036 scope.go:122] "RemoveContainer" containerID="3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099" Nov 01 22:29:25.493938 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:25.493909 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099\": container with ID starting with 3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099 not found: ID does not exist" containerID="3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099" Nov 01 22:29:25.494059 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.493945 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099"} err="failed to get container status \"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099\": rpc error: code = NotFound desc = could not find container \"3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099\": container with ID starting with 3cd07f654987e1ec3912269a8550cdc9e8cc7171717ab989367bad4f1b520099 not found: ID does not exist" Nov 01 22:29:25.539730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.538625 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:29:25.546664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.545088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:25.594533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.594492 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-pmk2p"] Nov 01 22:29:25.944296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.944257 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:29:25.945793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.945759 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:25.945938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:25.945824 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:26.339528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:26.339489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:26.445291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:26.445251 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" path="/var/lib/kubelet/pods/24c55a90-89bb-45c4-b44a-3e9400d86c46/volumes" Nov 01 22:29:26.445916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:26.445896 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" path="/var/lib/kubelet/pods/b4c5dadb-6e55-4c39-927a-3742c70b1584/volumes" Nov 01 22:29:26.946880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:26.946405 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:26.946880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:26.946492 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:27.514489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:27.511529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:27.945256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:27.945222 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:27.945435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:27.945293 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:28.136465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.136096 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:29:28.136465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.136357 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerName="maas-api" containerID="cri-o://e7d8caca4d22d428a16fa0fcb4150dbe95368872b42280ae8cf0ddaaf3383c8a" gracePeriod=30 Nov 01 22:29:28.164732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.164632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:28.328470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.312745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:28.499521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.476032 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:28.644483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.633936 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:28.644483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.644475 5036 generic.go:356] "Generic (PLEG): container finished" podID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerID="e7d8caca4d22d428a16fa0fcb4150dbe95368872b42280ae8cf0ddaaf3383c8a" exitCode=0 Nov 01 22:29:28.645101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.644624 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" event={"ID":"eec00664-bafb-4423-9c33-e154b9a4c5ba","Type":"ContainerDied","Data":"e7d8caca4d22d428a16fa0fcb4150dbe95368872b42280ae8cf0ddaaf3383c8a"} Nov 01 22:29:28.651467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.649488 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8"] Nov 01 22:29:28.706468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.692908 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a68ac-xn6l8"] Nov 01 22:29:28.821543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.787783 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.83:8443/health\": dial tcp 10.132.1.83:8443: connect: connection refused" Nov 01 22:29:28.821543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.805139 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:28.821862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.821541 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:29:28.970522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.948523 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:28.970522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.948590 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:28.971363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:28.971330 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:29.137905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.131058 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:29.166957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.166929 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:29:29.167282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.167258 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" containerID="cri-o://5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc" gracePeriod=30 Nov 01 22:29:29.308846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.302830 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:29.321230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.321196 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:29:29.573466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.572432 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\") pod \"eec00664-bafb-4423-9c33-e154b9a4c5ba\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " Nov 01 22:29:29.573466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.572539 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\" (UniqueName: \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\") pod \"eec00664-bafb-4423-9c33-e154b9a4c5ba\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " Nov 01 22:29:29.573466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.572675 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\") pod \"eec00664-bafb-4423-9c33-e154b9a4c5ba\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " Nov 01 22:29:29.573466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.572717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\") pod \"eec00664-bafb-4423-9c33-e154b9a4c5ba\" (UID: \"eec00664-bafb-4423-9c33-e154b9a4c5ba\") " Nov 01 22:29:29.604745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.575513 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca" pod "eec00664-bafb-4423-9c33-e154b9a4c5ba" (UID: "eec00664-bafb-4423-9c33-e154b9a4c5ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:29.633642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.633424 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls" pod "eec00664-bafb-4423-9c33-e154b9a4c5ba" (UID: "eec00664-bafb-4423-9c33-e154b9a4c5ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:29.633898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.633739 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w" pod "eec00664-bafb-4423-9c33-e154b9a4c5ba" (UID: "eec00664-bafb-4423-9c33-e154b9a4c5ba"). InnerVolumeSpecName "kube-api-access-67c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:29.637303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.637271 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs" pod "eec00664-bafb-4423-9c33-e154b9a4c5ba" (UID: "eec00664-bafb-4423-9c33-e154b9a4c5ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:29.658372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.658352 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" Nov 01 22:29:29.658818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.658355 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv" event={"ID":"eec00664-bafb-4423-9c33-e154b9a4c5ba","Type":"ContainerDied","Data":"32173950f126d7221493f502b79e2d442c4ad38d85f485183623d4bfd168d678"} Nov 01 22:29:29.658913 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.658867 5036 scope.go:122] "RemoveContainer" containerID="e7d8caca4d22d428a16fa0fcb4150dbe95368872b42280ae8cf0ddaaf3383c8a" Nov 01 22:29:29.683604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.681200 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:29.683604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.681231 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67c2w\" (UniqueName: \"kubernetes.io/projected/eec00664-bafb-4423-9c33-e154b9a4c5ba-kube-api-access-67c2w\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:29.683604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.681247 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec00664-bafb-4423-9c33-e154b9a4c5ba-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:29.683604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.681260 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec00664-bafb-4423-9c33-e154b9a4c5ba-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:29.695806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.688050 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:29:29.695806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.695090 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a68ac-74b5f54d95-hg9cv"] Nov 01 22:29:29.722683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.722656 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc"] Nov 01 22:29:29.723297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723284 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" Nov 01 22:29:29.723348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723298 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" Nov 01 22:29:29.723348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723312 5036 cpu_manager.go:414] "containerMap: removing container" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerName="maas-api" Nov 01 22:29:29.723348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723317 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerName="maas-api" Nov 01 22:29:29.723348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723325 5036 cpu_manager.go:414] "containerMap: removing container" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" Nov 01 22:29:29.723348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723330 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723375 5036 cpu_manager.go:414] "containerMap: removing container" podUID="04412c40-7832-4cdb-b2a7-6447fc47537e" containerName="revoke-keys" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723380 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04412c40-7832-4cdb-b2a7-6447fc47537e" containerName="revoke-keys" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723490 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="04412c40-7832-4cdb-b2a7-6447fc47537e" containerName="revoke-keys" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723501 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c5dadb-6e55-4c39-927a-3742c70b1584" containerName="istio-proxy" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723509 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" containerName="maas-api" Nov 01 22:29:29.723550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.723516 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c55a90-89bb-45c4-b44a-3e9400d86c46" containerName="istio-proxy" Nov 01 22:29:29.728684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.728667 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.732951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.732887 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:29:29.733590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.733075 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:29:29.736226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.736205 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc"] Nov 01 22:29:29.800468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.792328 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.800468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.792517 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qdr\" (UniqueName: \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.800468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.792590 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.898610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.898518 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.898941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.898917 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qdr\" (UniqueName: \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.899121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.899103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.914510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.913401 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.914510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.914157 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:29.916289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.916261 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:29.950489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.947338 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:29.950489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.947509 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:29.951597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:29.951570 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qdr\" (UniqueName: \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:30.046099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.046063 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:30.362574 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:29:30.362542 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e519d3e_9a87_4c0f_b82d_6b44ec05bcce.slice/crio-01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe WatchSource:0}: Error finding container 01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe: Status 404 returned error can't find the container with id 01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe Nov 01 22:29:30.363278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.363252 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc"] Nov 01 22:29:30.440841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.440814 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04412c40-7832-4cdb-b2a7-6447fc47537e" path="/var/lib/kubelet/pods/04412c40-7832-4cdb-b2a7-6447fc47537e/volumes" Nov 01 22:29:30.441318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.441302 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec00664-bafb-4423-9c33-e154b9a4c5ba" path="/var/lib/kubelet/pods/eec00664-bafb-4423-9c33-e154b9a4c5ba/volumes" Nov 01 22:29:30.810470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.802892 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" event={"ID":"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce","Type":"ContainerStarted","Data":"01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe"} Nov 01 22:29:30.944635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.944595 5036 patch_prober.go:29] interesting pod/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Nov 01 22:29:30.944811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:30.944659 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Nov 01 22:29:31.143479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:31.142598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:31.751660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:31.751626 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:31.871705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:31.870904 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" containerID="6d4f381603a8371d3a1e6a33f20954fd863c6b6a36c5055ebda72d3c37ee9d18" exitCode=0 Nov 01 22:29:31.871705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:31.871247 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" event={"ID":"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce","Type":"ContainerDied","Data":"6d4f381603a8371d3a1e6a33f20954fd863c6b6a36c5055ebda72d3c37ee9d18"} Nov 01 22:29:31.895708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:31.895672 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:32.089389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:32.089356 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:29:32.091406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:32.091307 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:29:32.257484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:32.256897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:32.421521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:32.417950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:32.937960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:32.937917 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:33.140933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.140856 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:33.214210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.214171 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\") pod \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " Nov 01 22:29:33.214412 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.214363 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\" (UniqueName: \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\") pod \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " Nov 01 22:29:33.214579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.214557 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\") pod \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\" (UID: \"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce\") " Nov 01 22:29:33.215055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.215019 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca" pod "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" (UID: "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:33.215302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.215280 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:33.218594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.217660 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr" pod "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" (UID: "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce"). InnerVolumeSpecName "kube-api-access-46qdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:33.218962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.218936 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca" pod "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" (UID: "8e519d3e-9a87-4c0f-b82d-6b44ec05bcce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:33.317228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.317182 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qdr\" (UniqueName: \"kubernetes.io/projected/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-kube-api-access-46qdr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:33.317228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.317219 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:33.851287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.851247 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:33.922467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.911435 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" event={"ID":"8e519d3e-9a87-4c0f-b82d-6b44ec05bcce","Type":"ContainerDied","Data":"01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe"} Nov 01 22:29:33.922467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.911580 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e6acec262ca30e71da74ebfbdaede1bda94cc23f84170e275ba360e7786dbe" Nov 01 22:29:33.922467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:33.911661 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc" Nov 01 22:29:34.467317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.467276 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:29:34.544900 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.544867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.544996 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545048 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545089 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545275 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545329 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545415 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.545846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.545664 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\") pod \"022493ad-82aa-4c71-b1f6-99a7a2437421\" (UID: \"022493ad-82aa-4c71-b1f6-99a7a2437421\") " Nov 01 22:29:34.546203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.546012 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:34.546907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.546360 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.546907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.546532 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:34.546907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.546877 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:34.547131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.546937 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:34.547131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.547077 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:34.548745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.548523 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:29:34.549146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.549093 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:34.549251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.549227 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "kube-api-access-f28j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:34.553109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.553080 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy" pod "022493ad-82aa-4c71-b1f6-99a7a2437421" (UID: "022493ad-82aa-4c71-b1f6-99a7a2437421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:34.648140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648099 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648139 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022493ad-82aa-4c71-b1f6-99a7a2437421-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648156 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648170 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648184 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28j5\" (UniqueName: \"kubernetes.io/projected/022493ad-82aa-4c71-b1f6-99a7a2437421-kube-api-access-f28j5\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648200 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648213 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.648380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.648226 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022493ad-82aa-4c71-b1f6-99a7a2437421-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:34.827662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.827622 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:34.926724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.926682 5036 generic.go:356] "Generic (PLEG): container finished" podID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerID="5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc" exitCode=0 Nov 01 22:29:34.926914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.926765 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" Nov 01 22:29:34.927041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.927002 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerDied","Data":"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc"} Nov 01 22:29:34.927129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.927069 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f" event={"ID":"022493ad-82aa-4c71-b1f6-99a7a2437421","Type":"ContainerDied","Data":"fa4aaeff44fae8a1b9e2d7d7d2fcab0858f613a56fdeb4688639365188749c3b"} Nov 01 22:29:34.927129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:34.927102 5036 scope.go:122] "RemoveContainer" containerID="5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc" Nov 01 22:29:35.050467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.047085 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:29:35.062661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.062274 5036 scope.go:122] "RemoveContainer" containerID="629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8" Nov 01 22:29:35.095680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.088700 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a00c6a-openshift-default-7f74497cbc-9fl6f"] Nov 01 22:29:35.151836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.151797 5036 scope.go:122] "RemoveContainer" containerID="5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc" Nov 01 22:29:35.152302 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:35.152272 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc\": container with ID starting with 5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc not found: ID does not exist" containerID="5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc" Nov 01 22:29:35.152800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.152311 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc"} err="failed to get container status \"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc\": rpc error: code = NotFound desc = could not find container \"5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc\": container with ID starting with 5ea7c5ad1c3125a2a461e92a62f162f16662ad8d98a7cc910713d8740ef166cc not found: ID does not exist" Nov 01 22:29:35.152800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.152330 5036 scope.go:122] "RemoveContainer" containerID="629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8" Nov 01 22:29:35.152800 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:35.152731 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8\": container with ID starting with 629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8 not found: ID does not exist" containerID="629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8" Nov 01 22:29:35.152800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.152756 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8"} err="failed to get container status \"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8\": rpc error: code = NotFound desc = could not find container \"629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8\": container with ID starting with 629e9d986ed2426e2e13eadd43af2c8cd4bc87fcb714826a215d0080486aafe8 not found: ID does not exist" Nov 01 22:29:35.526403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.526367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:35.656829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.655487 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:35.787837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:35.787744 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:36.341142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:36.341109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:36.439396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:36.439358 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" path="/var/lib/kubelet/pods/022493ad-82aa-4c71-b1f6-99a7a2437421/volumes" Nov 01 22:29:36.460989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:36.460957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:36.660676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:36.660590 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:36.980955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:36.980873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:37.096785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.096615 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj"] Nov 01 22:29:37.097701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097671 5036 cpu_manager.go:414] "containerMap: removing container" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.097701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097700 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.097919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097740 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" containerName="revoke-keys" Nov 01 22:29:37.097919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097750 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" containerName="revoke-keys" Nov 01 22:29:37.097919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097796 5036 cpu_manager.go:414] "containerMap: removing container" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.097919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097804 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.098140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097963 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.098140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097982 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" containerName="revoke-keys" Nov 01 22:29:37.098140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.097999 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="022493ad-82aa-4c71-b1f6-99a7a2437421" containerName="istio-proxy" Nov 01 22:29:37.127513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.107881 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.141465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.132520 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zmz\" (UniqueName: \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.141465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.132812 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.141465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.133011 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.163483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.162516 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:37.163483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.163011 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:29:37.211634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.211595 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:29:37.254962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.234838 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zmz\" (UniqueName: \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.254962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.234988 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.254962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.235035 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.254962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.248527 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.276133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.276042 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.318547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.301721 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zmz\" (UniqueName: \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\") pod \"maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.318547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.302087 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:37.407803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.407755 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj"] Nov 01 22:29:37.699897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.699870 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj"] Nov 01 22:29:37.700226 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:29:37.699990 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb67df8c_35e8_4ff6_9498_871be03690e1.slice/crio-bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7 WatchSource:0}: Error finding container bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7: Status 404 returned error can't find the container with id bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7 Nov 01 22:29:37.711707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.711679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:37.957726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:37.957643 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" event={"ID":"db67df8c-35e8-4ff6-9498-871be03690e1","Type":"ContainerStarted","Data":"bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7"} Nov 01 22:29:38.538062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:38.538023 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:38.968562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:38.968525 5036 generic.go:356] "Generic (PLEG): container finished" podID="db67df8c-35e8-4ff6-9498-871be03690e1" containerID="75342b22d7539f14aaa6602a15ab8648d2b08b1f2215d913013221905401c162" exitCode=0 Nov 01 22:29:38.968999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:38.968611 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" event={"ID":"db67df8c-35e8-4ff6-9498-871be03690e1","Type":"ContainerDied","Data":"75342b22d7539f14aaa6602a15ab8648d2b08b1f2215d913013221905401c162"} Nov 01 22:29:39.264695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:39.250572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:39.941559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:39.941489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:40.070752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.068887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:40.368845 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.368809 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:40.429872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.429830 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\" (UniqueName: \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\") pod \"db67df8c-35e8-4ff6-9498-871be03690e1\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " Nov 01 22:29:40.430081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.430061 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\") pod \"db67df8c-35e8-4ff6-9498-871be03690e1\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " Nov 01 22:29:40.430183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.430127 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\") pod \"db67df8c-35e8-4ff6-9498-871be03690e1\" (UID: \"db67df8c-35e8-4ff6-9498-871be03690e1\") " Nov 01 22:29:40.430664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.430633 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca" pod "db67df8c-35e8-4ff6-9498-871be03690e1" (UID: "db67df8c-35e8-4ff6-9498-871be03690e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:40.433776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.433341 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz" pod "db67df8c-35e8-4ff6-9498-871be03690e1" (UID: "db67df8c-35e8-4ff6-9498-871be03690e1"). InnerVolumeSpecName "kube-api-access-g5zmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:40.433776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.434143 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca" pod "db67df8c-35e8-4ff6-9498-871be03690e1" (UID: "db67df8c-35e8-4ff6-9498-871be03690e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:40.535871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.535844 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:40.536015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.535962 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db67df8c-35e8-4ff6-9498-871be03690e1-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:40.536015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.535988 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zmz\" (UniqueName: \"kubernetes.io/projected/db67df8c-35e8-4ff6-9498-871be03690e1-kube-api-access-g5zmz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:40.638381 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.638289 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:40.990277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.990187 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" Nov 01 22:29:40.990277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.990209 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj" event={"ID":"db67df8c-35e8-4ff6-9498-871be03690e1","Type":"ContainerDied","Data":"bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7"} Nov 01 22:29:40.990277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:40.990270 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd29f105077a91e61bd6796cfa910be9ff9a1b7d31fecf611a5c11d1182c8bc7" Nov 01 22:29:41.281728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:41.281689 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:42.068470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:42.065896 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:42.861875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:42.861836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:43.735993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:43.735957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:44.395024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:44.394984 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:45.054422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:45.054389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:45.936379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:45.936343 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:46.671468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:46.666898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:47.435783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:47.435698 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:48.340703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:48.340666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:48.859413 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:29:48.859368 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:29:49.118214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:49.118113 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:49.259713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:49.259673 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:49.385214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:49.385121 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:49.997476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:49.991073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:50.753284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:50.753241 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:51.306791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:51.306756 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:29:51.478582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:51.478546 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:52.106905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.106866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:52.287465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.287041 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:52.428519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.424665 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:52.623128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.623089 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc"] Nov 01 22:29:52.745373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.745276 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e3890e-fwqhc"] Nov 01 22:29:52.792047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.792006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:52.928057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:52.928009 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:53.163653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:53.163618 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:53.567629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:53.567597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:54.239227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:54.239186 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:54.440559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:54.440525 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e519d3e-9a87-4c0f-b82d-6b44ec05bcce" path="/var/lib/kubelet/pods/8e519d3e-9a87-4c0f-b82d-6b44ec05bcce/volumes" Nov 01 22:29:55.048835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:55.048800 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:55.771656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:55.771604 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:56.431661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:56.431621 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.010910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.010675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.142531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.142429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.277468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.275841 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.422487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.420633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.489821 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.470207 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj"] Nov 01 22:29:57.511753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.511717 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-db715b-qcjjj"] Nov 01 22:29:57.626889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.626808 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:57.909399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:57.909299 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:58.440608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:58.440574 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db67df8c-35e8-4ff6-9498-871be03690e1" path="/var/lib/kubelet/pods/db67df8c-35e8-4ff6-9498-871be03690e1/volumes" Nov 01 22:29:58.525427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:58.525396 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:59.340199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.340027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:59.466307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.466275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:59.578291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.578254 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:29:59.584535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.584494 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" containerID="cri-o://bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926" gracePeriod=30 Nov 01 22:29:59.609270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.609196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:59.961657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.961581 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq"] Nov 01 22:29:59.962261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.962245 5036 cpu_manager.go:414] "containerMap: removing container" podUID="db67df8c-35e8-4ff6-9498-871be03690e1" containerName="revoke-keys" Nov 01 22:29:59.962311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.962263 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db67df8c-35e8-4ff6-9498-871be03690e1" containerName="revoke-keys" Nov 01 22:29:59.962374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.962365 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="db67df8c-35e8-4ff6-9498-871be03690e1" containerName="revoke-keys" Nov 01 22:29:59.967492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.967438 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:29:59.975924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.975896 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:29:59.976055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.976020 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:29:59.976243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:29:59.976229 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq"] Nov 01 22:30:00.045849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.045815 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.046024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.045866 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.046147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.046122 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdcm\" (UniqueName: \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.136962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.136927 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:00.141909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.141885 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:00.142061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.142011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.146142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.146121 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:00.146265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.146238 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.147201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.147175 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdcm\" (UniqueName: \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.147332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.147314 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.147402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.147358 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.148148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.148121 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.150565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.150538 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.155771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.155753 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:00.155976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.155830 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.161721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.161579 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.161910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.161885 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:00.162031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.161937 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:00.164682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.164658 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:00.177076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.177040 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdcm\" (UniqueName: \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.188543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.188517 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:00.202748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.202598 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.209292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.208583 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:00.214858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.214423 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.229604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.229473 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:00.237423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.236766 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:00.244309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.243007 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:00.250788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.250762 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.251137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.251114 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.251465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.251428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4tl\" (UniqueName: \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.252518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.252494 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxrv\" (UniqueName: \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.257530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.256581 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.257530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.256619 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dqn\" (UniqueName: \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.257530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.256800 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4hs\" (UniqueName: \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.257530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.256906 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.257530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.254575 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:00.266676 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.266651 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:00.269598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.269577 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.272157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.272086 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:00.272263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.272169 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.276082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.275505 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:00.284758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.284735 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:00.358016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.357981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshmd\" (UniqueName: \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.358208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358034 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.358208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358192 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnd4\" (UniqueName: \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.358341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358246 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4hs\" (UniqueName: \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.358494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358428 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.358589 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358549 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gv85\" (UniqueName: \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.358645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx949\" (UniqueName: \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.358705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358665 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.358755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358728 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.358814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358783 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.358814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358801 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4tl\" (UniqueName: \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.358937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358832 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.358937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358868 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxrv\" (UniqueName: \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.358937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358883 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.358937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358900 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.358937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.358916 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dqn\" (UniqueName: \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.359990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.359964 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.360313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.360292 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.360675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.360655 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.360774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.360704 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.373749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.373462 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dqn\" (UniqueName: \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.374180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.374152 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4hs\" (UniqueName: \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.393494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.392439 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxrv\" (UniqueName: \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.393494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.393199 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4tl\" (UniqueName: \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.460701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460665 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv85\" (UniqueName: \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.460701 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460715 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx949\" (UniqueName: \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.460993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460846 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.460993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.460993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460940 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshmd\" (UniqueName: \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.460993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.460969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.462243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.461073 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnd4\" (UniqueName: \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.462243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.461152 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.462243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.462016 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.462554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.462525 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.462662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.462545 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.462662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.462601 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.474516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.474432 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv85\" (UniqueName: \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.474634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.474600 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshmd\" (UniqueName: \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.475228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.475208 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx949\" (UniqueName: \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.475526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.475508 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnd4\" (UniqueName: \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.511256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.511225 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:00.526702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.526669 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:00.540037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.540016 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:00.552294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.552264 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:00.567369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.567169 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:00.575498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.575422 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:00.612411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.612174 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:00.623160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.622840 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq"] Nov 01 22:30:00.633661 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:00.633621 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5800e07_03da_487d_b13a_06be45139c34.slice/crio-edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366 WatchSource:0}: Error finding container edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366: Status 404 returned error can't find the container with id edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366 Nov 01 22:30:00.638598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.638411 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:00.706513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.706429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:00.852975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.852939 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:00.865472 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:00.865095 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb955fd1_9d4f_4cbc_80ab_1ceecddf5676.slice/crio-5c9ffa1e2f52d4d3de6defc54de9151a866301dc27dfc7d30d74efbff0b6ccd7 WatchSource:0}: Error finding container 5c9ffa1e2f52d4d3de6defc54de9151a866301dc27dfc7d30d74efbff0b6ccd7: Status 404 returned error can't find the container with id 5c9ffa1e2f52d4d3de6defc54de9151a866301dc27dfc7d30d74efbff0b6ccd7 Nov 01 22:30:00.902496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.901514 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:00.909748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.909716 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:00.974247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:00.974199 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:01.010609 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:01.010572 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3e134e_b19e_4ddd_9510_58f9dc1886ee.slice/crio-90c40dac5f9cdf67a58bf9c51d49da9472b7c2141ee1a6b5cec16444e2388d40 WatchSource:0}: Error finding container 90c40dac5f9cdf67a58bf9c51d49da9472b7c2141ee1a6b5cec16444e2388d40: Status 404 returned error can't find the container with id 90c40dac5f9cdf67a58bf9c51d49da9472b7c2141ee1a6b5cec16444e2388d40 Nov 01 22:30:01.102924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.102880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:01.126290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.126260 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:01.140355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.139257 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:01.144685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.144661 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:01.181289 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:01.181254 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaa8f2c_815a_45c1_894e_33575fb052c3.slice/crio-1f5f1f57a3478b9632fcd78a3b88bf31fee1dcd9e20ac101d3e82c093a5f5916 WatchSource:0}: Error finding container 1f5f1f57a3478b9632fcd78a3b88bf31fee1dcd9e20ac101d3e82c093a5f5916: Status 404 returned error can't find the container with id 1f5f1f57a3478b9632fcd78a3b88bf31fee1dcd9e20ac101d3e82c093a5f5916 Nov 01 22:30:01.182626 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:01.182602 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded713ef_d135_45b8_b340_56ee615dacef.slice/crio-a7d7f20291b2b078e178115f2a5ac6ede715e22c2b0ae92bb22f0f63ce237efe WatchSource:0}: Error finding container a7d7f20291b2b078e178115f2a5ac6ede715e22c2b0ae92bb22f0f63ce237efe: Status 404 returned error can't find the container with id a7d7f20291b2b078e178115f2a5ac6ede715e22c2b0ae92bb22f0f63ce237efe Nov 01 22:30:01.265967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.265935 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:01.281996 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:01.281887 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ba743d_0024_4c94_a8b4_ab3595878e89.slice/crio-6cf061e7bddc3a4f61d326afbe55ce8cd7ee9bbdecfa148f9d36d53daa2021d0 WatchSource:0}: Error finding container 6cf061e7bddc3a4f61d326afbe55ce8cd7ee9bbdecfa148f9d36d53daa2021d0: Status 404 returned error can't find the container with id 6cf061e7bddc3a4f61d326afbe55ce8cd7ee9bbdecfa148f9d36d53daa2021d0 Nov 01 22:30:01.291497 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:01.289561 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c9a78_178c_4570_9b9b_8be21e21c88d.slice/crio-1ae09a670011d8c57f6fad009f735d0f7d423dffa768efdca126828b9993ef8d WatchSource:0}: Error finding container 1ae09a670011d8c57f6fad009f735d0f7d423dffa768efdca126828b9993ef8d: Status 404 returned error can't find the container with id 1ae09a670011d8c57f6fad009f735d0f7d423dffa768efdca126828b9993ef8d Nov 01 22:30:01.370170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.370135 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerStarted","Data":"1f5f1f57a3478b9632fcd78a3b88bf31fee1dcd9e20ac101d3e82c093a5f5916"} Nov 01 22:30:01.371742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.371704 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerStarted","Data":"1ae09a670011d8c57f6fad009f735d0f7d423dffa768efdca126828b9993ef8d"} Nov 01 22:30:01.372923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.372901 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerStarted","Data":"5c9ffa1e2f52d4d3de6defc54de9151a866301dc27dfc7d30d74efbff0b6ccd7"} Nov 01 22:30:01.374164 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.374141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerStarted","Data":"6cf061e7bddc3a4f61d326afbe55ce8cd7ee9bbdecfa148f9d36d53daa2021d0"} Nov 01 22:30:01.376341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.376294 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerStarted","Data":"a7d7f20291b2b078e178115f2a5ac6ede715e22c2b0ae92bb22f0f63ce237efe"} Nov 01 22:30:01.377896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.377870 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerStarted","Data":"af063b87fc0d0d9df9493c62960dfb9338cd2eb0041a623ba9f2d19c5a80fedb"} Nov 01 22:30:01.379285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.379256 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerStarted","Data":"90c40dac5f9cdf67a58bf9c51d49da9472b7c2141ee1a6b5cec16444e2388d40"} Nov 01 22:30:01.381023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.380996 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" event={"ID":"f5800e07-03da-487d-b13a-06be45139c34","Type":"ContainerStarted","Data":"05894bf1b9ebaebb321f78523e2465f0f032430f700a15d30f9cbf8a4ed657d9"} Nov 01 22:30:01.381149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.381050 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" event={"ID":"f5800e07-03da-487d-b13a-06be45139c34","Type":"ContainerStarted","Data":"edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366"} Nov 01 22:30:01.383071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.383046 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerStarted","Data":"00d4a1d0a1f5c2fa256185b72947eccbf81883b783b4e2094a9edae16fce64e6"} Nov 01 22:30:01.402856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.400687 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" podStartSLOduration=2.400669802 podStartE2EDuration="2.400669802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.399144776 +0000 UTC m=+2755.092125264" watchObservedRunningTime="2026-11-01 22:30:01.400669802 +0000 UTC m=+2755.093650289" Nov 01 22:30:01.644988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:01.644949 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:02.401591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.401558 5036 generic.go:356] "Generic (PLEG): container finished" podID="f5800e07-03da-487d-b13a-06be45139c34" containerID="05894bf1b9ebaebb321f78523e2465f0f032430f700a15d30f9cbf8a4ed657d9" exitCode=0 Nov 01 22:30:02.401727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.401635 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" event={"ID":"f5800e07-03da-487d-b13a-06be45139c34","Type":"ContainerDied","Data":"05894bf1b9ebaebb321f78523e2465f0f032430f700a15d30f9cbf8a4ed657d9"} Nov 01 22:30:02.403889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.403864 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerID="c2399856c50d9a78edc42d77a8278d0ee3d0a6797eee9abce4066a55408dc946" exitCode=60 Nov 01 22:30:02.404169 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.403900 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerDied","Data":"c2399856c50d9a78edc42d77a8278d0ee3d0a6797eee9abce4066a55408dc946"} Nov 01 22:30:02.404308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.404278 5036 scope.go:122] "RemoveContainer" containerID="c2399856c50d9a78edc42d77a8278d0ee3d0a6797eee9abce4066a55408dc946" Nov 01 22:30:02.406345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.406323 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerStarted","Data":"e3db052d8e92c37ccb68eb6e86d030d643b8f99d6ba0cfbcb1401099df947508"} Nov 01 22:30:02.409420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.409397 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerStarted","Data":"e7ed7c18b67704e41a94b7f82f3ec96c73cdd4eff5a3cd8285e64cba27260d38"} Nov 01 22:30:02.411673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.411643 5036 generic.go:356] "Generic (PLEG): container finished" podID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerID="be9c4ff6e3372510cd3b5af3e7e11b3e69e3be19c23fea20b9ad4d07b6abf63b" exitCode=60 Nov 01 22:30:02.412475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.412455 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerDied","Data":"be9c4ff6e3372510cd3b5af3e7e11b3e69e3be19c23fea20b9ad4d07b6abf63b"} Nov 01 22:30:02.412772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.412755 5036 scope.go:122] "RemoveContainer" containerID="be9c4ff6e3372510cd3b5af3e7e11b3e69e3be19c23fea20b9ad4d07b6abf63b" Nov 01 22:30:02.414849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.414829 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerStarted","Data":"aeb7df7c6a7d31247738d0873a1022fbd66692494cfc896b299456b2d74fd1c5"} Nov 01 22:30:02.416975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.416951 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerStarted","Data":"af9a07e81a9e91f802aa9feb16f9a0a4237ba62c7bad8c96740c69acebacf190"} Nov 01 22:30:02.426392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.426057 5036 generic.go:356] "Generic (PLEG): container finished" podID="147c71d2-3cad-44b5-b308-c18c113f666c" containerID="cf074ba2fb13193e1041977199272aee2e8e282b8ddf26fbc2640e529c4940da" exitCode=60 Nov 01 22:30:02.426392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.426152 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerDied","Data":"cf074ba2fb13193e1041977199272aee2e8e282b8ddf26fbc2640e529c4940da"} Nov 01 22:30:02.426581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.426496 5036 scope.go:122] "RemoveContainer" containerID="cf074ba2fb13193e1041977199272aee2e8e282b8ddf26fbc2640e529c4940da" Nov 01 22:30:02.429045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.429024 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:02.429511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.429490 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerStarted","Data":"967fe8e766679c5cda64c657c40a75ad88d4e719f342d1faff97582b97baeb23"} Nov 01 22:30:02.513668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.513609 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" podStartSLOduration=2.513587918 podStartE2EDuration="2.513587918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:02.462864396 +0000 UTC m=+2756.155844879" watchObservedRunningTime="2026-11-01 22:30:02.513587918 +0000 UTC m=+2756.206568399" Nov 01 22:30:02.541506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.532109 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" podStartSLOduration=2.532090281 podStartE2EDuration="2.532090281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:02.525839478 +0000 UTC m=+2756.218819968" watchObservedRunningTime="2026-11-01 22:30:02.532090281 +0000 UTC m=+2756.225070768" Nov 01 22:30:02.563287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.563240 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" podStartSLOduration=2.563228357 podStartE2EDuration="2.563228357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:02.562354884 +0000 UTC m=+2756.255335366" watchObservedRunningTime="2026-11-01 22:30:02.563228357 +0000 UTC m=+2756.256208843" Nov 01 22:30:02.601579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.601521 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" podStartSLOduration=2.601504588 podStartE2EDuration="2.601504588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:02.599554223 +0000 UTC m=+2756.292534710" watchObservedRunningTime="2026-11-01 22:30:02.601504588 +0000 UTC m=+2756.294485075" Nov 01 22:30:02.644408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:02.644362 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" podStartSLOduration=2.64434992 podStartE2EDuration="2.64434992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:02.642071573 +0000 UTC m=+2756.335052059" watchObservedRunningTime="2026-11-01 22:30:02.64434992 +0000 UTC m=+2756.337330405" Nov 01 22:30:03.028605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.028569 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x"] Nov 01 22:30:03.033497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.033469 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb31daa-kqt2x"] Nov 01 22:30:03.359560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.359475 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:03.443082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.443041 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerStarted","Data":"a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016"} Nov 01 22:30:03.445266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.445190 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerID="e3db052d8e92c37ccb68eb6e86d030d643b8f99d6ba0cfbcb1401099df947508" exitCode=60 Nov 01 22:30:03.445766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.445266 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerDied","Data":"e3db052d8e92c37ccb68eb6e86d030d643b8f99d6ba0cfbcb1401099df947508"} Nov 01 22:30:03.445766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.445559 5036 scope.go:122] "RemoveContainer" containerID="e3db052d8e92c37ccb68eb6e86d030d643b8f99d6ba0cfbcb1401099df947508" Nov 01 22:30:03.447702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.447677 5036 generic.go:356] "Generic (PLEG): container finished" podID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerID="e7ed7c18b67704e41a94b7f82f3ec96c73cdd4eff5a3cd8285e64cba27260d38" exitCode=60 Nov 01 22:30:03.447815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.447762 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerDied","Data":"e7ed7c18b67704e41a94b7f82f3ec96c73cdd4eff5a3cd8285e64cba27260d38"} Nov 01 22:30:03.448038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.448017 5036 scope.go:122] "RemoveContainer" containerID="e7ed7c18b67704e41a94b7f82f3ec96c73cdd4eff5a3cd8285e64cba27260d38" Nov 01 22:30:03.450104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.450078 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerStarted","Data":"cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37"} Nov 01 22:30:03.453655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.453628 5036 generic.go:356] "Generic (PLEG): container finished" podID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerID="aeb7df7c6a7d31247738d0873a1022fbd66692494cfc896b299456b2d74fd1c5" exitCode=60 Nov 01 22:30:03.453775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.453704 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerDied","Data":"aeb7df7c6a7d31247738d0873a1022fbd66692494cfc896b299456b2d74fd1c5"} Nov 01 22:30:03.454047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.454026 5036 scope.go:122] "RemoveContainer" containerID="aeb7df7c6a7d31247738d0873a1022fbd66692494cfc896b299456b2d74fd1c5" Nov 01 22:30:03.457523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.456976 5036 generic.go:356] "Generic (PLEG): container finished" podID="ded713ef-d135-45b8-b340-56ee615dacef" containerID="af9a07e81a9e91f802aa9feb16f9a0a4237ba62c7bad8c96740c69acebacf190" exitCode=60 Nov 01 22:30:03.457523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.457056 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerDied","Data":"af9a07e81a9e91f802aa9feb16f9a0a4237ba62c7bad8c96740c69acebacf190"} Nov 01 22:30:03.457523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.457369 5036 scope.go:122] "RemoveContainer" containerID="af9a07e81a9e91f802aa9feb16f9a0a4237ba62c7bad8c96740c69acebacf190" Nov 01 22:30:03.460233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.460215 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerStarted","Data":"76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507"} Nov 01 22:30:03.462294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.462271 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerID="967fe8e766679c5cda64c657c40a75ad88d4e719f342d1faff97582b97baeb23" exitCode=60 Nov 01 22:30:03.462566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.462543 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerDied","Data":"967fe8e766679c5cda64c657c40a75ad88d4e719f342d1faff97582b97baeb23"} Nov 01 22:30:03.462791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.462749 5036 scope.go:122] "RemoveContainer" containerID="967fe8e766679c5cda64c657c40a75ad88d4e719f342d1faff97582b97baeb23" Nov 01 22:30:03.464742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.464697 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" podStartSLOduration=3.464685554 podStartE2EDuration="3.464685554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:03.461504998 +0000 UTC m=+2757.154485485" watchObservedRunningTime="2026-11-01 22:30:03.464685554 +0000 UTC m=+2757.157666040" Nov 01 22:30:03.513656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.513612 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" podStartSLOduration=3.513596731 podStartE2EDuration="3.513596731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:03.505837072 +0000 UTC m=+2757.198817558" watchObservedRunningTime="2026-11-01 22:30:03.513596731 +0000 UTC m=+2757.206577216" Nov 01 22:30:03.586227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.586150 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" podStartSLOduration=3.5861322639999997 podStartE2EDuration="3.586132264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:03.582209021 +0000 UTC m=+2757.275189508" watchObservedRunningTime="2026-11-01 22:30:03.586132264 +0000 UTC m=+2757.279112752" Nov 01 22:30:03.683466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.682892 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:03.746500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.746462 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\") pod \"f5800e07-03da-487d-b13a-06be45139c34\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " Nov 01 22:30:03.746812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.746789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\" (UniqueName: \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\") pod \"f5800e07-03da-487d-b13a-06be45139c34\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " Nov 01 22:30:03.746861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.746837 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\") pod \"f5800e07-03da-487d-b13a-06be45139c34\" (UID: \"f5800e07-03da-487d-b13a-06be45139c34\") " Nov 01 22:30:03.746896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.746875 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca" pod "f5800e07-03da-487d-b13a-06be45139c34" (UID: "f5800e07-03da-487d-b13a-06be45139c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:03.747383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.747358 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5800e07-03da-487d-b13a-06be45139c34-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:03.782363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.782146 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm" pod "f5800e07-03da-487d-b13a-06be45139c34" (UID: "f5800e07-03da-487d-b13a-06be45139c34"). InnerVolumeSpecName "kube-api-access-fqdcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:03.782363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.782280 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca" pod "f5800e07-03da-487d-b13a-06be45139c34" (UID: "f5800e07-03da-487d-b13a-06be45139c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:03.849165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.849128 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdcm\" (UniqueName: \"kubernetes.io/projected/f5800e07-03da-487d-b13a-06be45139c34-kube-api-access-fqdcm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:03.849165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:03.849161 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5800e07-03da-487d-b13a-06be45139c34-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:04.292969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.292929 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:04.441026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.440988 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaa966a-64d9-403a-9485-b5f2d280b33a" path="/var/lib/kubelet/pods/ecaa966a-64d9-403a-9485-b5f2d280b33a/volumes" Nov 01 22:30:04.474153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.474127 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" Nov 01 22:30:04.474309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.474167 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq" event={"ID":"f5800e07-03da-487d-b13a-06be45139c34","Type":"ContainerDied","Data":"edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366"} Nov 01 22:30:04.474309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.474219 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf5b5ba532515c8bc2c83428c302f93dce439c83b803f5b1e878c2acc3b6366" Nov 01 22:30:04.476438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.476416 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerID="a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016" exitCode=60 Nov 01 22:30:04.476608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.476533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerDied","Data":"a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016"} Nov 01 22:30:04.476608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.476582 5036 scope.go:122] "RemoveContainer" containerID="c2399856c50d9a78edc42d77a8278d0ee3d0a6797eee9abce4066a55408dc946" Nov 01 22:30:04.477315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.476794 5036 scope.go:122] "RemoveContainer" containerID="a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016" Nov 01 22:30:04.477315 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:04.477097 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm_odh-ai-gateway-infra(8e2f4291-84e7-4287-b56e-ae20bd691777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" Nov 01 22:30:04.483857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.483831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerStarted","Data":"e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d"} Nov 01 22:30:04.486390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.486362 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerStarted","Data":"86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913"} Nov 01 22:30:04.489600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.489578 5036 generic.go:356] "Generic (PLEG): container finished" podID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerID="cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37" exitCode=60 Nov 01 22:30:04.489702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.489675 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerDied","Data":"cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37"} Nov 01 22:30:04.489961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.489946 5036 scope.go:122] "RemoveContainer" containerID="cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37" Nov 01 22:30:04.490175 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:04.490150 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz_odh-ai-gateway-infra(bb955fd1-9d4f-4cbc-80ab-1ceecddf5676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" Nov 01 22:30:04.494890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.494859 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerStarted","Data":"ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b"} Nov 01 22:30:04.498647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.498268 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerStarted","Data":"bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687"} Nov 01 22:30:04.507600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.507578 5036 scope.go:122] "RemoveContainer" containerID="be9c4ff6e3372510cd3b5af3e7e11b3e69e3be19c23fea20b9ad4d07b6abf63b" Nov 01 22:30:04.510136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.510106 5036 generic.go:356] "Generic (PLEG): container finished" podID="147c71d2-3cad-44b5-b308-c18c113f666c" containerID="76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507" exitCode=60 Nov 01 22:30:04.510254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.510201 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerDied","Data":"76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507"} Nov 01 22:30:04.510566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.510547 5036 scope.go:122] "RemoveContainer" containerID="76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507" Nov 01 22:30:04.510823 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:04.510799 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd_odh-ai-gateway-infra(147c71d2-3cad-44b5-b308-c18c113f666c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" Nov 01 22:30:04.512962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.512940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerStarted","Data":"9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0"} Nov 01 22:30:04.526693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:04.526669 5036 scope.go:122] "RemoveContainer" containerID="cf074ba2fb13193e1041977199272aee2e8e282b8ddf26fbc2640e529c4940da" Nov 01 22:30:05.150514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.145882 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:30:05.213956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.213918 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214242 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214408 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214496 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214614 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214665 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214710 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214779 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.214918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.214903 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\") pod \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\" (UID: \"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d\") " Nov 01 22:30:05.215759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.215730 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:05.216147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.216123 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:05.216759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.216734 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:05.217341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.217314 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:05.217797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.217772 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:05.219820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.219799 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:05.219940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.219839 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "kube-api-access-7k255". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:05.220157 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.220135 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:05.221119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.221093 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo" pod "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" (UID: "c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:30:05.317182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317148 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317185 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k255\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-kube-api-access-7k255\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317204 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317218 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317231 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317245 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317260 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317274 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.317643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.317287 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.376861 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.376823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:05.525878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.525839 5036 generic.go:356] "Generic (PLEG): container finished" podID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerID="86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913" exitCode=60 Nov 01 22:30:05.526062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.525956 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerDied","Data":"86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913"} Nov 01 22:30:05.526062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.526045 5036 scope.go:122] "RemoveContainer" containerID="e7ed7c18b67704e41a94b7f82f3ec96c73cdd4eff5a3cd8285e64cba27260d38" Nov 01 22:30:05.526392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.526371 5036 scope.go:122] "RemoveContainer" containerID="86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913" Nov 01 22:30:05.526825 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.526790 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k_odh-ai-gateway-infra(f89c9a78-178c-4570-9b9b-8be21e21c88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" Nov 01 22:30:05.528092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.528074 5036 scope.go:122] "RemoveContainer" containerID="cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37" Nov 01 22:30:05.528350 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.528328 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz_odh-ai-gateway-infra(bb955fd1-9d4f-4cbc-80ab-1ceecddf5676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" Nov 01 22:30:05.529996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.529972 5036 generic.go:356] "Generic (PLEG): container finished" podID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerID="ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b" exitCode=60 Nov 01 22:30:05.530114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.530038 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerDied","Data":"ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b"} Nov 01 22:30:05.530267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.530253 5036 scope.go:122] "RemoveContainer" containerID="ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b" Nov 01 22:30:05.530541 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.530509 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc_odh-ai-gateway-infra(88ba743d-0024-4c94-a8b4-ab3595878e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" Nov 01 22:30:05.532587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.532564 5036 generic.go:356] "Generic (PLEG): container finished" podID="ded713ef-d135-45b8-b340-56ee615dacef" containerID="bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687" exitCode=60 Nov 01 22:30:05.532702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.532648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerDied","Data":"bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687"} Nov 01 22:30:05.532869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.532853 5036 scope.go:122] "RemoveContainer" containerID="bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687" Nov 01 22:30:05.533111 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.533091 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx_odh-ai-gateway-infra(ded713ef-d135-45b8-b340-56ee615dacef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" podUID="ded713ef-d135-45b8-b340-56ee615dacef" Nov 01 22:30:05.534513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.534495 5036 scope.go:122] "RemoveContainer" containerID="76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507" Nov 01 22:30:05.534754 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.534728 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd_odh-ai-gateway-infra(147c71d2-3cad-44b5-b308-c18c113f666c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" Nov 01 22:30:05.536406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.536385 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerID="9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0" exitCode=60 Nov 01 22:30:05.536533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.536470 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerDied","Data":"9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0"} Nov 01 22:30:05.536707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.536693 5036 scope.go:122] "RemoveContainer" containerID="9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0" Nov 01 22:30:05.536909 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.536892 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj_odh-ai-gateway-infra(6d3e134e-b19e-4ddd-9510-58f9dc1886ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" Nov 01 22:30:05.538566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.538547 5036 scope.go:122] "RemoveContainer" containerID="a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016" Nov 01 22:30:05.538845 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.538819 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm_odh-ai-gateway-infra(8e2f4291-84e7-4287-b56e-ae20bd691777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" Nov 01 22:30:05.540434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.540412 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerID="e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d" exitCode=60 Nov 01 22:30:05.540527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.540438 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerDied","Data":"e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d"} Nov 01 22:30:05.540830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.540807 5036 scope.go:122] "RemoveContainer" containerID="e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d" Nov 01 22:30:05.541050 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.541024 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz_odh-ai-gateway-infra(3aaa8f2c-815a-45c1-894e-33575fb052c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" Nov 01 22:30:05.542076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.542055 5036 generic.go:356] "Generic (PLEG): container finished" podID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerID="bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926" exitCode=0 Nov 01 22:30:05.542163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.542096 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" Nov 01 22:30:05.542163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.542106 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" event={"ID":"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d","Type":"ContainerDied","Data":"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926"} Nov 01 22:30:05.542163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.542141 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq" event={"ID":"c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d","Type":"ContainerDied","Data":"dccaa351ffeb542fcf2ccb2562518aa4468b6c258aa62a7f94dda52a89d10c7e"} Nov 01 22:30:05.543819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.543801 5036 scope.go:122] "RemoveContainer" containerID="aeb7df7c6a7d31247738d0873a1022fbd66692494cfc896b299456b2d74fd1c5" Nov 01 22:30:05.559804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.559783 5036 scope.go:122] "RemoveContainer" containerID="af9a07e81a9e91f802aa9feb16f9a0a4237ba62c7bad8c96740c69acebacf190" Nov 01 22:30:05.574571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.574552 5036 scope.go:122] "RemoveContainer" containerID="967fe8e766679c5cda64c657c40a75ad88d4e719f342d1faff97582b97baeb23" Nov 01 22:30:05.591067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.591046 5036 scope.go:122] "RemoveContainer" containerID="e3db052d8e92c37ccb68eb6e86d030d643b8f99d6ba0cfbcb1401099df947508" Nov 01 22:30:05.591848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.591793 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:05.640664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.638925 5036 scope.go:122] "RemoveContainer" containerID="bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926" Nov 01 22:30:05.656967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.656945 5036 scope.go:122] "RemoveContainer" containerID="bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926" Nov 01 22:30:05.657368 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:05.657338 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926\": container with ID starting with bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926 not found: ID does not exist" containerID="bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926" Nov 01 22:30:05.657494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.657379 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926"} err="failed to get container status \"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926\": rpc error: code = NotFound desc = could not find container \"bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926\": container with ID starting with bd9c8d798c7aa88499fbe09cb6667a79a44de8bcfcb83cf76667d6ba6c8ac926 not found: ID does not exist" Nov 01 22:30:05.730424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.730378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:05.742710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.742270 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:30:05.748209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.748175 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-fg4gq"] Nov 01 22:30:05.854684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.854212 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:30:05.855133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.854854 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" containerID="cri-o://7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9" gracePeriod=30 Nov 01 22:30:05.895709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:05.895583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.121660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.121573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.282477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.281996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.392514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.392363 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m"] Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.393505 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f5800e07-03da-487d-b13a-06be45139c34" containerName="revoke-keys" Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.393532 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5800e07-03da-487d-b13a-06be45139c34" containerName="revoke-keys" Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.393567 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.393576 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.395388 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5800e07-03da-487d-b13a-06be45139c34" containerName="revoke-keys" Nov 01 22:30:06.409382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.395425 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" containerName="istio-proxy" Nov 01 22:30:06.413918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.413889 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.415400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.415366 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m"] Nov 01 22:30:06.421085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.421061 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:06.434278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.433873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.455859 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.455835 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d" path="/var/lib/kubelet/pods/c76cc9f5-fbd4-41b1-83b1-c9a68bd1d93d/volumes" Nov 01 22:30:06.460863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.460828 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.461022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.460927 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.461266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.461240 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvlm\" (UniqueName: \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.553141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.553115 5036 scope.go:122] "RemoveContainer" containerID="e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d" Nov 01 22:30:06.553539 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:06.553511 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz_odh-ai-gateway-infra(3aaa8f2c-815a-45c1-894e-33575fb052c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" Nov 01 22:30:06.555946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.555926 5036 scope.go:122] "RemoveContainer" containerID="86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913" Nov 01 22:30:06.556255 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:06.556234 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k_odh-ai-gateway-infra(f89c9a78-178c-4570-9b9b-8be21e21c88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" Nov 01 22:30:06.557892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.557870 5036 scope.go:122] "RemoveContainer" containerID="ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b" Nov 01 22:30:06.558200 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:06.558171 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc_odh-ai-gateway-infra(88ba743d-0024-4c94-a8b4-ab3595878e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" Nov 01 22:30:06.559596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.559576 5036 scope.go:122] "RemoveContainer" containerID="bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687" Nov 01 22:30:06.559836 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:06.559816 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx_odh-ai-gateway-infra(ded713ef-d135-45b8-b340-56ee615dacef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" podUID="ded713ef-d135-45b8-b340-56ee615dacef" Nov 01 22:30:06.561290 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.561272 5036 scope.go:122] "RemoveContainer" containerID="9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0" Nov 01 22:30:06.561525 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:06.561506 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj_odh-ai-gateway-infra(6d3e134e-b19e-4ddd-9510-58f9dc1886ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" Nov 01 22:30:06.562188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.562157 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.562323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.562222 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.562323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.562308 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvlm\" (UniqueName: \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.562927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.562905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.564739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.564716 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.576951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.576920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvlm\" (UniqueName: \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:06.741741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:06.741664 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:07.030785 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:07.030756 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6473c3f9_5872_4228_b7c9_2b11f7f5003f.slice/crio-f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8 WatchSource:0}: Error finding container f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8: Status 404 returned error can't find the container with id f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8 Nov 01 22:30:07.030953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:07.030931 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m"] Nov 01 22:30:07.341574 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:07.337904 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:07.585832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:07.585742 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" event={"ID":"6473c3f9-5872-4228-b7c9-2b11f7f5003f","Type":"ContainerStarted","Data":"4f424a3309ef4ba4e66683fd37d1601bf53f6ddbeaf9c94f9f0ecc94ef3c7ee5"} Nov 01 22:30:07.585832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:07.585831 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" event={"ID":"6473c3f9-5872-4228-b7c9-2b11f7f5003f","Type":"ContainerStarted","Data":"f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8"} Nov 01 22:30:07.611925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:07.611868 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" podStartSLOduration=1.611856241 podStartE2EDuration="1.611856241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:07.605868379 +0000 UTC m=+2761.298848866" watchObservedRunningTime="2026-11-01 22:30:07.611856241 +0000 UTC m=+2761.304836727" Nov 01 22:30:08.038433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:08.038396 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:08.597163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:08.597113 5036 generic.go:356] "Generic (PLEG): container finished" podID="6473c3f9-5872-4228-b7c9-2b11f7f5003f" containerID="4f424a3309ef4ba4e66683fd37d1601bf53f6ddbeaf9c94f9f0ecc94ef3c7ee5" exitCode=0 Nov 01 22:30:08.597731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:08.597357 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" event={"ID":"6473c3f9-5872-4228-b7c9-2b11f7f5003f","Type":"ContainerDied","Data":"4f424a3309ef4ba4e66683fd37d1601bf53f6ddbeaf9c94f9f0ecc94ef3c7ee5"} Nov 01 22:30:08.710192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:08.701710 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.388467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.385285 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.795547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.795518 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:09.830269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.830233 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\" (UniqueName: \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\") pod \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " Nov 01 22:30:09.830472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.830292 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\") pod \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " Nov 01 22:30:09.830472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.830370 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\") pod \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\" (UID: \"6473c3f9-5872-4228-b7c9-2b11f7f5003f\") " Nov 01 22:30:09.830826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.830799 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca" pod "6473c3f9-5872-4228-b7c9-2b11f7f5003f" (UID: "6473c3f9-5872-4228-b7c9-2b11f7f5003f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:09.831143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.831115 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:09.832851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.832824 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm" pod "6473c3f9-5872-4228-b7c9-2b11f7f5003f" (UID: "6473c3f9-5872-4228-b7c9-2b11f7f5003f"). InnerVolumeSpecName "kube-api-access-mdvlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:09.837483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.833483 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca" pod "6473c3f9-5872-4228-b7c9-2b11f7f5003f" (UID: "6473c3f9-5872-4228-b7c9-2b11f7f5003f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:09.933004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.932928 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvlm\" (UniqueName: \"kubernetes.io/projected/6473c3f9-5872-4228-b7c9-2b11f7f5003f-kube-api-access-mdvlm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:09.933004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:09.932959 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6473c3f9-5872-4228-b7c9-2b11f7f5003f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:10.046827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.046782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:10.216601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.212254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:10.390475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.387117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:10.620992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.620955 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" Nov 01 22:30:10.620992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.620972 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m" event={"ID":"6473c3f9-5872-4228-b7c9-2b11f7f5003f","Type":"ContainerDied","Data":"f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8"} Nov 01 22:30:10.621247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.621024 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f44bc9bf55b42994cd7ce233912e2dca8b259bd0b83d8483ce948019ecd603c8" Nov 01 22:30:10.893961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:10.893932 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.169078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.169053 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:30:11.276522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276605 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276638 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276696 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276797 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276856 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.276987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.276932 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.277043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.277102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277083 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\") pod \"4f28f366-969c-4428-b4d8-9956897cc5da\" (UID: \"4f28f366-969c-4428-b4d8-9956897cc5da\") " Nov 01 22:30:11.277489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277415 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:11.277624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277526 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:11.277692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277667 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:11.277796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:11.277965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277946 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.278058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277971 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.278058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.277986 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.278058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.278001 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f28f366-969c-4428-b4d8-9956897cc5da-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.278329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.278173 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:11.279720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.279543 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:30:11.279832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.279767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "kube-api-access-vbbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:11.280099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.280078 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:11.280220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.280203 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token" pod "4f28f366-969c-4428-b4d8-9956897cc5da" (UID: "4f28f366-969c-4428-b4d8-9956897cc5da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:11.384284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.380782 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbpq\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-kube-api-access-vbbpq\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.384284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.380815 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f28f366-969c-4428-b4d8-9956897cc5da-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.384284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.380832 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.384284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.380846 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f28f366-969c-4428-b4d8-9956897cc5da-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.384284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.380859 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f28f366-969c-4428-b4d8-9956897cc5da-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.468144 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.467705 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.676201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.676106 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.766461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.765777 5036 generic.go:356] "Generic (PLEG): container finished" podID="4f28f366-969c-4428-b4d8-9956897cc5da" containerID="7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9" exitCode=0 Nov 01 22:30:11.766461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.766062 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerDied","Data":"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9"} Nov 01 22:30:11.766461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.766130 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" event={"ID":"4f28f366-969c-4428-b4d8-9956897cc5da","Type":"ContainerDied","Data":"36d44bd63a584fc70d63d01a3db5d13e8bb15b75cbcc9376f1fbef305e2e66da"} Nov 01 22:30:11.766461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.766164 5036 scope.go:122] "RemoveContainer" containerID="7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9" Nov 01 22:30:11.766461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.766407 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb" Nov 01 22:30:11.825001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.821633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.872035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.869593 5036 scope.go:122] "RemoveContainer" containerID="71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2" Nov 01 22:30:11.908770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.906939 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq"] Nov 01 22:30:11.908770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.907175 5036 scope.go:122] "RemoveContainer" containerID="7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9" Nov 01 22:30:11.908770 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:11.908071 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9\": container with ID starting with 7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9 not found: ID does not exist" containerID="7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9" Nov 01 22:30:11.908770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.908108 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9"} err="failed to get container status \"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9\": rpc error: code = NotFound desc = could not find container \"7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9\": container with ID starting with 7d82002f655cffe97b9ce453e58352eb77ba3fba79aa81e9b4024c72e4bc54c9 not found: ID does not exist" Nov 01 22:30:11.908770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.908135 5036 scope.go:122] "RemoveContainer" containerID="71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2" Nov 01 22:30:11.909419 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:11.908793 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2\": container with ID starting with 71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2 not found: ID does not exist" containerID="71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2" Nov 01 22:30:11.909419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.908833 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2"} err="failed to get container status \"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2\": rpc error: code = NotFound desc = could not find container \"71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2\": container with ID starting with 71d06d098e1ec4ab8ae91943a2434f056e932af421f684c712bb3a8396326cf2 not found: ID does not exist" Nov 01 22:30:11.913644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.913614 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-51e3b7-j9rtq"] Nov 01 22:30:11.926002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.925926 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:30:11.935490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.935391 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-db715b-openshift-default-d756b6495-dntjb"] Nov 01 22:30:11.966758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:11.966559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:12.335979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:12.335673 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:12.444756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:12.444716 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" path="/var/lib/kubelet/pods/4f28f366-969c-4428-b4d8-9956897cc5da/volumes" Nov 01 22:30:12.445280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:12.445263 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5800e07-03da-487d-b13a-06be45139c34" path="/var/lib/kubelet/pods/f5800e07-03da-487d-b13a-06be45139c34/volumes" Nov 01 22:30:12.828358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:12.828314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:13.546569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:13.546531 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:14.345274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:14.345224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:14.432467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:14.431838 5036 scope.go:122] "RemoveContainer" containerID="cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37" Nov 01 22:30:14.806728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:14.806696 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerStarted","Data":"8cbc375f04508e3e60fd0e875b40dd9fa30da84de54ad78e81f64d554e34d9ff"} Nov 01 22:30:15.039903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.039868 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:15.743090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.742989 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:15.836094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.836053 5036 generic.go:356] "Generic (PLEG): container finished" podID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerID="8cbc375f04508e3e60fd0e875b40dd9fa30da84de54ad78e81f64d554e34d9ff" exitCode=60 Nov 01 22:30:15.836584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.836122 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerDied","Data":"8cbc375f04508e3e60fd0e875b40dd9fa30da84de54ad78e81f64d554e34d9ff"} Nov 01 22:30:15.836584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.836216 5036 scope.go:122] "RemoveContainer" containerID="cbbba188885f16d2a6f67066836c9bd4e732b0f687d55bc6105363ed68f05c37" Nov 01 22:30:15.836584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.836565 5036 scope.go:122] "RemoveContainer" containerID="8cbc375f04508e3e60fd0e875b40dd9fa30da84de54ad78e81f64d554e34d9ff" Nov 01 22:30:15.836935 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:15.836913 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz_odh-ai-gateway-infra(bb955fd1-9d4f-4cbc-80ab-1ceecddf5676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" Nov 01 22:30:15.839568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:15.839544 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:16.436801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:16.436769 5036 scope.go:122] "RemoveContainer" containerID="76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507" Nov 01 22:30:16.436988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:16.436919 5036 scope.go:122] "RemoveContainer" containerID="9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0" Nov 01 22:30:16.437071 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:16.437002 5036 scope.go:122] "RemoveContainer" containerID="e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d" Nov 01 22:30:16.621477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:16.621045 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:16.786483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:16.786339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:17.175718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.175690 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:17.313879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.313838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\" (UniqueName: \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\") pod \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " Nov 01 22:30:17.314119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.314076 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\") pod \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\" (UID: \"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676\") " Nov 01 22:30:17.314862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.314831 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca" pod "bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" (UID: "bb955fd1-9d4f-4cbc-80ab-1ceecddf5676"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:17.316764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.316732 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn" pod "bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" (UID: "bb955fd1-9d4f-4cbc-80ab-1ceecddf5676"). InnerVolumeSpecName "kube-api-access-45dqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:17.416016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.415981 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dqn\" (UniqueName: \"kubernetes.io/projected/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-kube-api-access-45dqn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:17.416016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.416012 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:17.430682 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.430658 5036 scope.go:122] "RemoveContainer" containerID="86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913" Nov 01 22:30:17.430904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.430884 5036 scope.go:122] "RemoveContainer" containerID="a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016" Nov 01 22:30:17.569954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.569886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:17.925630 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.925595 5036 generic.go:356] "Generic (PLEG): container finished" podID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerID="c06b3945318c3cc172cc268414a24d1b391086ab8ce185238d51bb3ae8096b6d" exitCode=60 Nov 01 22:30:17.925818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.925671 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerDied","Data":"c06b3945318c3cc172cc268414a24d1b391086ab8ce185238d51bb3ae8096b6d"} Nov 01 22:30:17.925818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.925745 5036 scope.go:122] "RemoveContainer" containerID="e17b4f8f31d9e9d45e83319df46c8c709212b1e26c9b35e0766093df9403c10d" Nov 01 22:30:17.926238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.926211 5036 scope.go:122] "RemoveContainer" containerID="c06b3945318c3cc172cc268414a24d1b391086ab8ce185238d51bb3ae8096b6d" Nov 01 22:30:17.926667 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:17.926638 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz_odh-ai-gateway-infra(3aaa8f2c-815a-45c1-894e-33575fb052c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" Nov 01 22:30:17.927795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.927771 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" Nov 01 22:30:17.928390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.928363 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz" event={"ID":"bb955fd1-9d4f-4cbc-80ab-1ceecddf5676","Type":"ContainerDied","Data":"5c9ffa1e2f52d4d3de6defc54de9151a866301dc27dfc7d30d74efbff0b6ccd7"} Nov 01 22:30:17.931264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.931240 5036 generic.go:356] "Generic (PLEG): container finished" podID="147c71d2-3cad-44b5-b308-c18c113f666c" containerID="3f8563d3fc4ad349551220bd457c1aae4d0fad8944a810ed1681d813c526897b" exitCode=60 Nov 01 22:30:17.931359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.931314 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerDied","Data":"3f8563d3fc4ad349551220bd457c1aae4d0fad8944a810ed1681d813c526897b"} Nov 01 22:30:17.931763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.931743 5036 scope.go:122] "RemoveContainer" containerID="3f8563d3fc4ad349551220bd457c1aae4d0fad8944a810ed1681d813c526897b" Nov 01 22:30:17.932040 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:17.932021 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd_odh-ai-gateway-infra(147c71d2-3cad-44b5-b308-c18c113f666c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" Nov 01 22:30:17.933844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.933816 5036 generic.go:356] "Generic (PLEG): container finished" podID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerID="ea044e87b4a484495a347df7ad548fffdc66a8965eef3dd45f5cd2668590b169" exitCode=60 Nov 01 22:30:17.933973 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.933864 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerDied","Data":"ea044e87b4a484495a347df7ad548fffdc66a8965eef3dd45f5cd2668590b169"} Nov 01 22:30:17.934237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.934214 5036 scope.go:122] "RemoveContainer" containerID="ea044e87b4a484495a347df7ad548fffdc66a8965eef3dd45f5cd2668590b169" Nov 01 22:30:17.934574 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:17.934549 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj_odh-ai-gateway-infra(6d3e134e-b19e-4ddd-9510-58f9dc1886ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" Nov 01 22:30:17.941793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.941777 5036 scope.go:122] "RemoveContainer" containerID="8cbc375f04508e3e60fd0e875b40dd9fa30da84de54ad78e81f64d554e34d9ff" Nov 01 22:30:17.953029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.953007 5036 scope.go:122] "RemoveContainer" containerID="76f941b91c7f1d8d68e3c32a8cab6b6f20c9ae796a8f71f528a899fc2d10b507" Nov 01 22:30:17.962482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.962426 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:17.967935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.967913 5036 scope.go:122] "RemoveContainer" containerID="9733bbd2e0c60df16c7ba2d83a8869b8cc29877ed29d0dea952fd73cddaeeec0" Nov 01 22:30:17.974511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:17.974486 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e2bf63-29892870-c27mz"] Nov 01 22:30:18.098534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.098431 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:18.236590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.236556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:18.245359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.245326 5036 scope.go:122] "RemoveContainer" containerID="7200acd54af5075b451f8281dbfb167c0ad39e35d1754d7170af35018d59f9ed" Nov 01 22:30:18.262317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.262284 5036 scope.go:122] "RemoveContainer" containerID="e4709177a2355ffe8f6f3d7bab964ab6228060d3f0e65402845b4b74b78fad4f" Nov 01 22:30:18.346726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.346652 5036 scope.go:122] "RemoveContainer" containerID="7ba925241730cba3a2c56d0975678c99fb0bb97227dca4f01463af15cd19f638" Nov 01 22:30:18.462551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.460130 5036 scope.go:122] "RemoveContainer" containerID="bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687" Nov 01 22:30:18.462551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.461737 5036 scope.go:122] "RemoveContainer" containerID="ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b" Nov 01 22:30:18.558061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.558033 5036 scope.go:122] "RemoveContainer" containerID="8cfc6b1089b27f84b563df0b44511cc0ce2a856964bcb5a5222c96cd910faea3" Nov 01 22:30:18.566238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.566210 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" path="/var/lib/kubelet/pods/bb955fd1-9d4f-4cbc-80ab-1ceecddf5676/volumes" Nov 01 22:30:18.583639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.583545 5036 scope.go:122] "RemoveContainer" containerID="1cfc38831a58f45c88c298ad5db3ad2c39a376bafdbb5cc84a8f20c4d15b2a21" Nov 01 22:30:18.732992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.732934 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:18.860698 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:18.860665 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:30:18.945679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.945650 5036 generic.go:356] "Generic (PLEG): container finished" podID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerID="ff4e267d82a7b2fcd07ca3bc18dc98723e735bd00245da08b85ac1eb6d7e92d2" exitCode=60 Nov 01 22:30:18.945790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.945689 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerDied","Data":"ff4e267d82a7b2fcd07ca3bc18dc98723e735bd00245da08b85ac1eb6d7e92d2"} Nov 01 22:30:18.945790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.945756 5036 scope.go:122] "RemoveContainer" containerID="86e9eaba6d70598735143a6e647a465c715aa540468d8b736e663e495057f913" Nov 01 22:30:18.946224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.946200 5036 scope.go:122] "RemoveContainer" containerID="ff4e267d82a7b2fcd07ca3bc18dc98723e735bd00245da08b85ac1eb6d7e92d2" Nov 01 22:30:18.946663 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:18.946627 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k_odh-ai-gateway-infra(f89c9a78-178c-4570-9b9b-8be21e21c88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" Nov 01 22:30:18.951613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.951590 5036 generic.go:356] "Generic (PLEG): container finished" podID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerID="886075c6fb600aad034234c1dff5ba9155346fe3f26e7d968f339cea9e34f7bf" exitCode=60 Nov 01 22:30:18.951706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.951653 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerDied","Data":"886075c6fb600aad034234c1dff5ba9155346fe3f26e7d968f339cea9e34f7bf"} Nov 01 22:30:18.952047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.952022 5036 scope.go:122] "RemoveContainer" containerID="886075c6fb600aad034234c1dff5ba9155346fe3f26e7d968f339cea9e34f7bf" Nov 01 22:30:18.952354 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:18.952334 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm_odh-ai-gateway-infra(8e2f4291-84e7-4287-b56e-ae20bd691777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" Nov 01 22:30:18.958596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.958574 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:18.961067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:18.961044 5036 scope.go:122] "RemoveContainer" containerID="a84c8196d92afe0a38fa65fe38ade382a4ee258bae81bbdc84402724365fe016" Nov 01 22:30:19.023489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.023434 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:19.046600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.046121 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:19.251732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.251666 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:19.374837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.374809 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:19.387882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.387855 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:19.458688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.458640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\") pod \"147c71d2-3cad-44b5-b308-c18c113f666c\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " Nov 01 22:30:19.458946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.458698 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\" (UniqueName: \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\") pod \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " Nov 01 22:30:19.458946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.458732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\" (UniqueName: \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\") pod \"147c71d2-3cad-44b5-b308-c18c113f666c\" (UID: \"147c71d2-3cad-44b5-b308-c18c113f666c\") " Nov 01 22:30:19.458946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.458853 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\") pod \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\" (UID: \"6d3e134e-b19e-4ddd-9510-58f9dc1886ee\") " Nov 01 22:30:19.459255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.458947 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\" (UniqueName: \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\") pod \"3aaa8f2c-815a-45c1-894e-33575fb052c3\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " Nov 01 22:30:19.459255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.459020 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\") pod \"3aaa8f2c-815a-45c1-894e-33575fb052c3\" (UID: \"3aaa8f2c-815a-45c1-894e-33575fb052c3\") " Nov 01 22:30:19.460044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.460012 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca" pod "3aaa8f2c-815a-45c1-894e-33575fb052c3" (UID: "3aaa8f2c-815a-45c1-894e-33575fb052c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:19.460167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.460137 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca" pod "147c71d2-3cad-44b5-b308-c18c113f666c" (UID: "147c71d2-3cad-44b5-b308-c18c113f666c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:19.461591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.461558 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca" pod "6d3e134e-b19e-4ddd-9510-58f9dc1886ee" (UID: "6d3e134e-b19e-4ddd-9510-58f9dc1886ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:19.462993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.462966 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl" pod "6d3e134e-b19e-4ddd-9510-58f9dc1886ee" (UID: "6d3e134e-b19e-4ddd-9510-58f9dc1886ee"). InnerVolumeSpecName "kube-api-access-xd4tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:19.466081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.466054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs" pod "147c71d2-3cad-44b5-b308-c18c113f666c" (UID: "147c71d2-3cad-44b5-b308-c18c113f666c"). InnerVolumeSpecName "kube-api-access-9g4hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:19.467349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.467324 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949" pod "3aaa8f2c-815a-45c1-894e-33575fb052c3" (UID: "3aaa8f2c-815a-45c1-894e-33575fb052c3"). InnerVolumeSpecName "kube-api-access-xx949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:19.561062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561024 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.561062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561062 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx949\" (UniqueName: \"kubernetes.io/projected/3aaa8f2c-815a-45c1-894e-33575fb052c3-kube-api-access-xx949\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.561348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561077 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa8f2c-815a-45c1-894e-33575fb052c3-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.561348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561090 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/147c71d2-3cad-44b5-b308-c18c113f666c-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.561348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561104 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4tl\" (UniqueName: \"kubernetes.io/projected/6d3e134e-b19e-4ddd-9510-58f9dc1886ee-kube-api-access-xd4tl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.561348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.561119 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4hs\" (UniqueName: \"kubernetes.io/projected/147c71d2-3cad-44b5-b308-c18c113f666c-kube-api-access-9g4hs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:19.618202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.618168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:19.743406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.743368 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:19.888534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.883504 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:19.909531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.905313 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:30:19.909531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.906148 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" containerID="cri-o://1b5328cb60a9b1c89282da01ff2f0a861af806afd2316ceb3bf88c21cae3afe5" gracePeriod=30 Nov 01 22:30:19.965726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.965700 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" Nov 01 22:30:19.965910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.965739 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz" event={"ID":"3aaa8f2c-815a-45c1-894e-33575fb052c3","Type":"ContainerDied","Data":"1f5f1f57a3478b9632fcd78a3b88bf31fee1dcd9e20ac101d3e82c093a5f5916"} Nov 01 22:30:19.965910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.965795 5036 scope.go:122] "RemoveContainer" containerID="c06b3945318c3cc172cc268414a24d1b391086ab8ce185238d51bb3ae8096b6d" Nov 01 22:30:19.970263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.970240 5036 generic.go:356] "Generic (PLEG): container finished" podID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerID="ee6fe7080db81c7aa5febc3b5bf5b6135e890be338d6be1ef83130dd1e9b1b3b" exitCode=60 Nov 01 22:30:19.970402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.970306 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerDied","Data":"ee6fe7080db81c7aa5febc3b5bf5b6135e890be338d6be1ef83130dd1e9b1b3b"} Nov 01 22:30:19.970718 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.970697 5036 scope.go:122] "RemoveContainer" containerID="ee6fe7080db81c7aa5febc3b5bf5b6135e890be338d6be1ef83130dd1e9b1b3b" Nov 01 22:30:19.971017 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:19.970993 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc_odh-ai-gateway-infra(88ba743d-0024-4c94-a8b4-ab3595878e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" Nov 01 22:30:19.972820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.972792 5036 generic.go:356] "Generic (PLEG): container finished" podID="ded713ef-d135-45b8-b340-56ee615dacef" containerID="8bdd6528920d835901ea7293c3880741d78884f7afefa97334f4b0744fafa99c" exitCode=60 Nov 01 22:30:19.972952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.972857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerDied","Data":"8bdd6528920d835901ea7293c3880741d78884f7afefa97334f4b0744fafa99c"} Nov 01 22:30:19.973226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.973205 5036 scope.go:122] "RemoveContainer" containerID="8bdd6528920d835901ea7293c3880741d78884f7afefa97334f4b0744fafa99c" Nov 01 22:30:19.973607 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:19.973511 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx_odh-ai-gateway-infra(ded713ef-d135-45b8-b340-56ee615dacef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" podUID="ded713ef-d135-45b8-b340-56ee615dacef" Nov 01 22:30:19.975543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.974751 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" Nov 01 22:30:19.975788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.974771 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd" event={"ID":"147c71d2-3cad-44b5-b308-c18c113f666c","Type":"ContainerDied","Data":"af063b87fc0d0d9df9493c62960dfb9338cd2eb0041a623ba9f2d19c5a80fedb"} Nov 01 22:30:19.976749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.976597 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" event={"ID":"6d3e134e-b19e-4ddd-9510-58f9dc1886ee","Type":"ContainerDied","Data":"90c40dac5f9cdf67a58bf9c51d49da9472b7c2141ee1a6b5cec16444e2388d40"} Nov 01 22:30:19.976749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.976674 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj" Nov 01 22:30:19.983817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.983799 5036 scope.go:122] "RemoveContainer" containerID="ac2d8b57392d124bf7cb386b96841d20aefaf15466cfe670ca4df156854f988b" Nov 01 22:30:19.985474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:19.985430 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:20.123518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.119924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:20.186818 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.186786 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:20.192039 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.190167 5036 scope.go:122] "RemoveContainer" containerID="bd4e4813ed5f37a1c1a7bff300e4c32ad88ec4eb0a76af6dde96b6232609c687" Nov 01 22:30:20.273513 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.273369 5036 scope.go:122] "RemoveContainer" containerID="3f8563d3fc4ad349551220bd457c1aae4d0fad8944a810ed1681d813c526897b" Nov 01 22:30:20.277750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.277717 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:20.288884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.288819 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892870-xd2vz"] Nov 01 22:30:20.294055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.294026 5036 scope.go:122] "RemoveContainer" containerID="ea044e87b4a484495a347df7ad548fffdc66a8965eef3dd45f5cd2668590b169" Nov 01 22:30:20.296863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.296840 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:20.304503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.304480 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-521734-29892870-h5wkd"] Nov 01 22:30:20.323794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.320563 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:20.326896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.325333 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892870-255sj"] Nov 01 22:30:20.341990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.341967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:20.465132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.465096 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" path="/var/lib/kubelet/pods/147c71d2-3cad-44b5-b308-c18c113f666c/volumes" Nov 01 22:30:20.466022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.465992 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" path="/var/lib/kubelet/pods/3aaa8f2c-815a-45c1-894e-33575fb052c3/volumes" Nov 01 22:30:20.466629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.466608 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" path="/var/lib/kubelet/pods/6d3e134e-b19e-4ddd-9510-58f9dc1886ee/volumes" Nov 01 22:30:20.487796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.487773 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:20.529479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.529432 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5"] Nov 01 22:30:20.530874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530837 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.530874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530863 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530882 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530890 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530900 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530905 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530917 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530922 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530934 5036 cpu_manager.go:414] "containerMap: removing container" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530939 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530952 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530960 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530974 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530981 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530990 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.530996 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531007 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531014 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531024 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6473c3f9-5872-4228-b7c9-2b11f7f5003f" containerName="revoke-keys" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531029 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6473c3f9-5872-4228-b7c9-2b11f7f5003f" containerName="revoke-keys" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531057 5036 cpu_manager.go:414] "containerMap: removing container" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531063 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531078 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531083 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531095 5036 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.531151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531099 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531197 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531206 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531215 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531222 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531229 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531234 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531239 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6473c3f9-5872-4228-b7c9-2b11f7f5003f" containerName="revoke-keys" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531245 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531251 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531256 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531263 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531269 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531274 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa8f2c-815a-45c1-894e-33575fb052c3" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531282 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531288 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.532690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.531293 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:20.535844 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.535823 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5"] Nov 01 22:30:20.535953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.535907 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.560431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.560409 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:20.592398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.592372 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:20.598373 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.598343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\" (UniqueName: \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\") pod \"8e2f4291-84e7-4287-b56e-ae20bd691777\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " Nov 01 22:30:20.598609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.598509 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\") pod \"f89c9a78-178c-4570-9b9b-8be21e21c88d\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " Nov 01 22:30:20.598697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.598680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\" (UniqueName: \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\") pod \"f89c9a78-178c-4570-9b9b-8be21e21c88d\" (UID: \"f89c9a78-178c-4570-9b9b-8be21e21c88d\") " Nov 01 22:30:20.598767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.598732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\") pod \"8e2f4291-84e7-4287-b56e-ae20bd691777\" (UID: \"8e2f4291-84e7-4287-b56e-ae20bd691777\") " Nov 01 22:30:20.599127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599097 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca" pod "f89c9a78-178c-4570-9b9b-8be21e21c88d" (UID: "f89c9a78-178c-4570-9b9b-8be21e21c88d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.599217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599152 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bj6\" (UniqueName: \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.599279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599223 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.599392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599368 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.599659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599638 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c9a78-178c-4570-9b9b-8be21e21c88d-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.599783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.599758 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca" pod "8e2f4291-84e7-4287-b56e-ae20bd691777" (UID: "8e2f4291-84e7-4287-b56e-ae20bd691777"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.603659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.603634 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv" pod "8e2f4291-84e7-4287-b56e-ae20bd691777" (UID: "8e2f4291-84e7-4287-b56e-ae20bd691777"). InnerVolumeSpecName "kube-api-access-7kxrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:20.603796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.603713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85" pod "f89c9a78-178c-4570-9b9b-8be21e21c88d" (UID: "f89c9a78-178c-4570-9b9b-8be21e21c88d"). InnerVolumeSpecName "kube-api-access-9gv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:20.701060 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.700969 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.701217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701173 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bj6\" (UniqueName: \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.701293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701231 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.701362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701313 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxrv\" (UniqueName: \"kubernetes.io/projected/8e2f4291-84e7-4287-b56e-ae20bd691777-kube-api-access-7kxrv\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.701362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701330 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gv85\" (UniqueName: \"kubernetes.io/projected/f89c9a78-178c-4570-9b9b-8be21e21c88d-kube-api-access-9gv85\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.701362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701345 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2f4291-84e7-4287-b56e-ae20bd691777-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.702002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.701977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.703378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.703353 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.713928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.713903 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bj6\" (UniqueName: \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\") pod \"maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:20.874614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.874580 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:21.000202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.998725 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" Nov 01 22:30:21.000202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.998774 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k" event={"ID":"f89c9a78-178c-4570-9b9b-8be21e21c88d","Type":"ContainerDied","Data":"1ae09a670011d8c57f6fad009f735d0f7d423dffa768efdca126828b9993ef8d"} Nov 01 22:30:21.000202 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:20.998838 5036 scope.go:122] "RemoveContainer" containerID="ff4e267d82a7b2fcd07ca3bc18dc98723e735bd00245da08b85ac1eb6d7e92d2" Nov 01 22:30:21.009862 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.009827 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" event={"ID":"8e2f4291-84e7-4287-b56e-ae20bd691777","Type":"ContainerDied","Data":"00d4a1d0a1f5c2fa256185b72947eccbf81883b783b4e2094a9edae16fce64e6"} Nov 01 22:30:21.009992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.009933 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm" Nov 01 22:30:21.022915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.022259 5036 scope.go:122] "RemoveContainer" containerID="886075c6fb600aad034234c1dff5ba9155346fe3f26e7d968f339cea9e34f7bf" Nov 01 22:30:21.093933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.093770 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:21.127815 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.127784 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:21.142288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.142256 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892870-2hn5k"] Nov 01 22:30:21.145160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.145117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:21.163755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.163728 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:21.175886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.175484 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d2832c-29892870-2sdnm"] Nov 01 22:30:21.182191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.182162 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5"] Nov 01 22:30:21.212182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.212134 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:21.262875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.262648 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:21.310214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.310175 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\" (UniqueName: \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\") pod \"88ba743d-0024-4c94-a8b4-ab3595878e89\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " Nov 01 22:30:21.310670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.310344 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\") pod \"88ba743d-0024-4c94-a8b4-ab3595878e89\" (UID: \"88ba743d-0024-4c94-a8b4-ab3595878e89\") " Nov 01 22:30:21.312552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.312521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca" pod "88ba743d-0024-4c94-a8b4-ab3595878e89" (UID: "88ba743d-0024-4c94-a8b4-ab3595878e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:21.316656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.315729 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ba743d-0024-4c94-a8b4-ab3595878e89-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.323109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.323070 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd" pod "88ba743d-0024-4c94-a8b4-ab3595878e89" (UID: "88ba743d-0024-4c94-a8b4-ab3595878e89"). InnerVolumeSpecName "kube-api-access-dshmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:21.376113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.376081 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:21.417987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.417952 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\") pod \"ded713ef-d135-45b8-b340-56ee615dacef\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " Nov 01 22:30:21.418274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.418254 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\" (UniqueName: \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\") pod \"ded713ef-d135-45b8-b340-56ee615dacef\" (UID: \"ded713ef-d135-45b8-b340-56ee615dacef\") " Nov 01 22:30:21.418377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.418350 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca" pod "ded713ef-d135-45b8-b340-56ee615dacef" (UID: "ded713ef-d135-45b8-b340-56ee615dacef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:21.418867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.418839 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshmd\" (UniqueName: \"kubernetes.io/projected/88ba743d-0024-4c94-a8b4-ab3595878e89-kube-api-access-dshmd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.418980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.418869 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded713ef-d135-45b8-b340-56ee615dacef-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.421191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.421163 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4" pod "ded713ef-d135-45b8-b340-56ee615dacef" (UID: "ded713ef-d135-45b8-b340-56ee615dacef"). InnerVolumeSpecName "kube-api-access-chnd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:21.520423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:21.520387 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnd4\" (UniqueName: \"kubernetes.io/projected/ded713ef-d135-45b8-b340-56ee615dacef-kube-api-access-chnd4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:22.030884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.030841 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" event={"ID":"88ba743d-0024-4c94-a8b4-ab3595878e89","Type":"ContainerDied","Data":"6cf061e7bddc3a4f61d326afbe55ce8cd7ee9bbdecfa148f9d36d53daa2021d0"} Nov 01 22:30:22.030884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.030873 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc" Nov 01 22:30:22.031147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.030941 5036 scope.go:122] "RemoveContainer" containerID="ee6fe7080db81c7aa5febc3b5bf5b6135e890be338d6be1ef83130dd1e9b1b3b" Nov 01 22:30:22.032564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.032536 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" event={"ID":"ded713ef-d135-45b8-b340-56ee615dacef","Type":"ContainerDied","Data":"a7d7f20291b2b078e178115f2a5ac6ede715e22c2b0ae92bb22f0f63ce237efe"} Nov 01 22:30:22.032755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.032546 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx" Nov 01 22:30:22.039505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.039435 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" event={"ID":"554bc699-0b01-4212-92ae-5716cb2ad921","Type":"ContainerStarted","Data":"469f1cd66485670fa8a6ab5aaacbd79a13354a61679bfb1249bea4601635a18d"} Nov 01 22:30:22.039633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.039533 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" event={"ID":"554bc699-0b01-4212-92ae-5716cb2ad921","Type":"ContainerStarted","Data":"28ef56efc2a435eb70cc0f700f4a0c5419a51e06b659c770cac2832024e92ce0"} Nov 01 22:30:22.044526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.044429 5036 scope.go:122] "RemoveContainer" containerID="8bdd6528920d835901ea7293c3880741d78884f7afefa97334f4b0744fafa99c" Nov 01 22:30:22.064341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.064292 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" podStartSLOduration=2.064278355 podStartE2EDuration="2.064278355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:22.060819427 +0000 UTC m=+2775.753799916" watchObservedRunningTime="2026-11-01 22:30:22.064278355 +0000 UTC m=+2775.757258839" Nov 01 22:30:22.079759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.079718 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:22.084051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.084026 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892870-fdkfc"] Nov 01 22:30:22.092983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.092960 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:22.096898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.096876 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892870-mnwpx"] Nov 01 22:30:22.279374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.279338 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:22.439414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.439336 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" path="/var/lib/kubelet/pods/88ba743d-0024-4c94-a8b4-ab3595878e89/volumes" Nov 01 22:30:22.439834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.439811 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" path="/var/lib/kubelet/pods/8e2f4291-84e7-4287-b56e-ae20bd691777/volumes" Nov 01 22:30:22.440207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.440193 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded713ef-d135-45b8-b340-56ee615dacef" path="/var/lib/kubelet/pods/ded713ef-d135-45b8-b340-56ee615dacef/volumes" Nov 01 22:30:22.440923 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:22.440904 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" path="/var/lib/kubelet/pods/f89c9a78-178c-4570-9b9b-8be21e21c88d/volumes" Nov 01 22:30:23.055152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.055112 5036 generic.go:356] "Generic (PLEG): container finished" podID="554bc699-0b01-4212-92ae-5716cb2ad921" containerID="469f1cd66485670fa8a6ab5aaacbd79a13354a61679bfb1249bea4601635a18d" exitCode=0 Nov 01 22:30:23.055362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.055206 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" event={"ID":"554bc699-0b01-4212-92ae-5716cb2ad921","Type":"ContainerDied","Data":"469f1cd66485670fa8a6ab5aaacbd79a13354a61679bfb1249bea4601635a18d"} Nov 01 22:30:23.139986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.139941 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:23.856850 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.856807 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:30:23.857270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.856865 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:30:23.868742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:23.868706 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:24.258468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.258424 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:24.355958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.355926 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\" (UniqueName: \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\") pod \"554bc699-0b01-4212-92ae-5716cb2ad921\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " Nov 01 22:30:24.356125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.355968 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\") pod \"554bc699-0b01-4212-92ae-5716cb2ad921\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " Nov 01 22:30:24.356125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.356113 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\") pod \"554bc699-0b01-4212-92ae-5716cb2ad921\" (UID: \"554bc699-0b01-4212-92ae-5716cb2ad921\") " Nov 01 22:30:24.356516 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.356482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca" pod "554bc699-0b01-4212-92ae-5716cb2ad921" (UID: "554bc699-0b01-4212-92ae-5716cb2ad921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:24.358074 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.358052 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6" pod "554bc699-0b01-4212-92ae-5716cb2ad921" (UID: "554bc699-0b01-4212-92ae-5716cb2ad921"). InnerVolumeSpecName "kube-api-access-d2bj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:24.358246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.358226 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca" pod "554bc699-0b01-4212-92ae-5716cb2ad921" (UID: "554bc699-0b01-4212-92ae-5716cb2ad921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:24.457570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.457534 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:24.457570 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.457570 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bj6\" (UniqueName: \"kubernetes.io/projected/554bc699-0b01-4212-92ae-5716cb2ad921-kube-api-access-d2bj6\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:24.457740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.457581 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554bc699-0b01-4212-92ae-5716cb2ad921-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:24.670527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:24.670484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.077826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.077794 5036 generic.go:356] "Generic (PLEG): container finished" podID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerID="1b5328cb60a9b1c89282da01ff2f0a861af806afd2316ceb3bf88c21cae3afe5" exitCode=0 Nov 01 22:30:25.078210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.078071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" event={"ID":"95c44b2a-a745-4bca-9142-b05b85c4663f","Type":"ContainerDied","Data":"1b5328cb60a9b1c89282da01ff2f0a861af806afd2316ceb3bf88c21cae3afe5"} Nov 01 22:30:25.079577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.079556 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" event={"ID":"554bc699-0b01-4212-92ae-5716cb2ad921","Type":"ContainerDied","Data":"28ef56efc2a435eb70cc0f700f4a0c5419a51e06b659c770cac2832024e92ce0"} Nov 01 22:30:25.079699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.079593 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ef56efc2a435eb70cc0f700f4a0c5419a51e06b659c770cac2832024e92ce0" Nov 01 22:30:25.079699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.079646 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5" Nov 01 22:30:25.200190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.200159 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:30:25.268875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.268844 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.268959 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.268983 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269043 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269164 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:25.269356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269334 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269409 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269421 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:25.269472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269457 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:25.269666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269512 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269567 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:25.269666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269597 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.269666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.269653 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\") pod \"95c44b2a-a745-4bca-9142-b05b85c4663f\" (UID: \"95c44b2a-a745-4bca-9142-b05b85c4663f\") " Nov 01 22:30:25.270307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270131 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:25.270656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270554 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.270656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270583 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.270656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270602 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.270656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270616 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c44b2a-a745-4bca-9142-b05b85c4663f-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.270656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.270633 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.272206 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.272181 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:30:25.272735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.272632 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:25.272846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.272789 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:25.273790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.273766 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw" pod "95c44b2a-a745-4bca-9142-b05b85c4663f" (UID: "95c44b2a-a745-4bca-9142-b05b85c4663f"). InnerVolumeSpecName "kube-api-access-g52kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:25.372211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.372168 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.372211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.372210 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.372418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.372229 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c44b2a-a745-4bca-9142-b05b85c4663f-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.372418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.372244 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52kw\" (UniqueName: \"kubernetes.io/projected/95c44b2a-a745-4bca-9142-b05b85c4663f-kube-api-access-g52kw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:25.408326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.408279 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.533423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:25.533392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.107465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.101873 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" Nov 01 22:30:26.107465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.101910 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485" event={"ID":"95c44b2a-a745-4bca-9142-b05b85c4663f","Type":"ContainerDied","Data":"aa6bd5084707e99bac77b3c16c7f4b9c46a34cad055c37dd97fc59503e039212"} Nov 01 22:30:26.107465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.101984 5036 scope.go:122] "RemoveContainer" containerID="1b5328cb60a9b1c89282da01ff2f0a861af806afd2316ceb3bf88c21cae3afe5" Nov 01 22:30:26.109053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.109026 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.138786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.138613 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:30:26.143310 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.143286 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-bt485"] Nov 01 22:30:26.265468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.264946 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.418116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.408039 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.522463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.521942 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" path="/var/lib/kubelet/pods/95c44b2a-a745-4bca-9142-b05b85c4663f/volumes" Nov 01 22:30:26.522463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.522325 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m"] Nov 01 22:30:26.522463 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.522354 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-f758d9-sbx9m"] Nov 01 22:30:26.566654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.566604 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.862584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:26.862544 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:27.430378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:27.430339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:28.285531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:28.285495 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:28.440387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:28.440338 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6473c3f9-5872-4228-b7c9-2b11f7f5003f" path="/var/lib/kubelet/pods/6473c3f9-5872-4228-b7c9-2b11f7f5003f/volumes" Nov 01 22:30:28.930746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:28.930708 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:29.643795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:29.643763 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:30.566217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:30.566182 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:30.704931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:30.704895 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 22:30:30.725772 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:30.725742 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 22:30:30.823319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:30.823219 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:30.946584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:30.946545 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:31.651038 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:31.650999 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:32.569004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.568960 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:32.792713 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.792679 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv"] Nov 01 22:30:32.793323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793306 5036 cpu_manager.go:414] "containerMap: removing container" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" Nov 01 22:30:32.793323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793323 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793335 5036 cpu_manager.go:414] "containerMap: removing container" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793340 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793357 5036 cpu_manager.go:414] "containerMap: removing container" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793362 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147c71d2-3cad-44b5-b308-c18c113f666c" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793368 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793373 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793395 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793400 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793406 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793411 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89c9a78-178c-4570-9b9b-8be21e21c88d" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793425 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793430 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793454 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793460 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793470 5036 cpu_manager.go:414] "containerMap: removing container" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793474 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793482 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793486 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793494 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793499 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793507 5036 cpu_manager.go:414] "containerMap: removing container" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793512 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f28f366-969c-4428-b4d8-9956897cc5da" containerName="istio-proxy" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793520 5036 cpu_manager.go:414] "containerMap: removing container" podUID="554bc699-0b01-4212-92ae-5716cb2ad921" containerName="revoke-keys" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793524 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554bc699-0b01-4212-92ae-5716cb2ad921" containerName="revoke-keys" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793531 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:32.793569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793536 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793636 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793645 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793653 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793662 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="554bc699-0b01-4212-92ae-5716cb2ad921" containerName="revoke-keys" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793673 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3e134e-b19e-4ddd-9510-58f9dc1886ee" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793683 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c44b2a-a745-4bca-9142-b05b85c4663f" containerName="istio-proxy" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793689 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793696 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793704 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793711 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:32.795076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.793717 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb955fd1-9d4f-4cbc-80ab-1ceecddf5676" containerName="cleanup" Nov 01 22:30:32.799016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.798989 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:32.803456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.803403 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:30:32.803735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.803713 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:32.806689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.806662 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv"] Nov 01 22:30:32.948497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.948404 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:32.948661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.948597 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:32.948661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:32.948656 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscx4\" (UniqueName: \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.049961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.049917 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.050160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.050041 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.050160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.050103 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscx4\" (UniqueName: \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.050637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.050610 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.052464 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.052418 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.063196 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.063166 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscx4\" (UniqueName: \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.117756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.117728 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:33.406310 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:33.406272 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7102eb_079e_4410_96e3_8778901ce3b3.slice/crio-f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359 WatchSource:0}: Error finding container f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359: Status 404 returned error can't find the container with id f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359 Nov 01 22:30:33.408984 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.408952 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv"] Nov 01 22:30:33.461535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.461504 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:33.582404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.582362 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:33.793143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.793110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:33.945469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:33.940167 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:34.221958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.221865 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" event={"ID":"1a7102eb-079e-4410-96e3-8778901ce3b3","Type":"ContainerStarted","Data":"73357e963dee7ca777265db8b2ecc97b9639325323c3316edef0bbfa9fae2986"} Nov 01 22:30:34.221958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.221936 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" event={"ID":"1a7102eb-079e-4410-96e3-8778901ce3b3","Type":"ContainerStarted","Data":"f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359"} Nov 01 22:30:34.231529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.231504 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc"] Nov 01 22:30:34.232111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232098 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:34.232158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232113 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded713ef-d135-45b8-b340-56ee615dacef" containerName="cleanup" Nov 01 22:30:34.232158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232145 5036 cpu_manager.go:414] "containerMap: removing container" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:34.232158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232150 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:34.232267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232252 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ba743d-0024-4c94-a8b4-ab3595878e89" containerName="cleanup" Nov 01 22:30:34.232319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.232270 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2f4291-84e7-4287-b56e-ae20bd691777" containerName="cleanup" Nov 01 22:30:34.235909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.235892 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.245219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.245191 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc"] Nov 01 22:30:34.248072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.248035 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" podStartSLOduration=2.248022672 podStartE2EDuration="2.248022672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:34.241512737 +0000 UTC m=+2787.934493227" watchObservedRunningTime="2026-11-01 22:30:34.248022672 +0000 UTC m=+2787.941003184" Nov 01 22:30:34.374436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.374406 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.374657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.374642 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpnl\" (UniqueName: \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.374723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.374701 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.476748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.476672 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.476898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.476843 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.477024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.477007 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpnl\" (UniqueName: \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.477495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.477464 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.479347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.479323 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.493292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.493269 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpnl\" (UniqueName: \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.560328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.560297 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:34.725989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.725943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:34.852910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:34.852876 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc"] Nov 01 22:30:34.853119 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:34.853099 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf740c66_4784_4428_8b13_90db3b380c5a.slice/crio-0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890 WatchSource:0}: Error finding container 0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890: Status 404 returned error can't find the container with id 0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890 Nov 01 22:30:35.232968 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.232930 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" event={"ID":"df740c66-4784-4428-8b13-90db3b380c5a","Type":"ContainerStarted","Data":"5090c6f2fdfea8850a4098f0e48f4c3a287bb670f4634d4c41be60e0eb4e45fe"} Nov 01 22:30:35.233127 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.233015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" event={"ID":"df740c66-4784-4428-8b13-90db3b380c5a","Type":"ContainerStarted","Data":"0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890"} Nov 01 22:30:35.234669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.234642 5036 generic.go:356] "Generic (PLEG): container finished" podID="1a7102eb-079e-4410-96e3-8778901ce3b3" containerID="73357e963dee7ca777265db8b2ecc97b9639325323c3316edef0bbfa9fae2986" exitCode=0 Nov 01 22:30:35.234790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.234723 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" event={"ID":"1a7102eb-079e-4410-96e3-8778901ce3b3","Type":"ContainerDied","Data":"73357e963dee7ca777265db8b2ecc97b9639325323c3316edef0bbfa9fae2986"} Nov 01 22:30:35.264287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.264239 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" podStartSLOduration=1.2642243180000001 podStartE2EDuration="1.264224318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:35.260933132 +0000 UTC m=+2788.953913619" watchObservedRunningTime="2026-11-01 22:30:35.264224318 +0000 UTC m=+2788.957204804" Nov 01 22:30:35.660855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.660815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:35.709495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.708411 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" podUID="552811b9-12a3-45e2-a681-9640706a1fbf" containerName="payload-processing" containerID="cri-o://cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14" gracePeriod=25 Nov 01 22:30:35.726705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:35.726658 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" containerID="cri-o://86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629" gracePeriod=25 Nov 01 22:30:36.002121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.002090 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 22:30:36.103809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.103764 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\") pod \"552811b9-12a3-45e2-a681-9640706a1fbf\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " Nov 01 22:30:36.103981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.103963 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\" (UniqueName: \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\") pod \"552811b9-12a3-45e2-a681-9640706a1fbf\" (UID: \"552811b9-12a3-45e2-a681-9640706a1fbf\") " Nov 01 22:30:36.104160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.104125 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume" pod "552811b9-12a3-45e2-a681-9640706a1fbf" (UID: "552811b9-12a3-45e2-a681-9640706a1fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:36.104646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.104626 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552811b9-12a3-45e2-a681-9640706a1fbf-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:36.106298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.106272 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd" pod "552811b9-12a3-45e2-a681-9640706a1fbf" (UID: "552811b9-12a3-45e2-a681-9640706a1fbf"). InnerVolumeSpecName "kube-api-access-9llzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:36.152146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.152119 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf"] Nov 01 22:30:36.152829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.152812 5036 cpu_manager.go:414] "containerMap: removing container" podUID="552811b9-12a3-45e2-a681-9640706a1fbf" containerName="payload-processing" Nov 01 22:30:36.152917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.152830 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552811b9-12a3-45e2-a681-9640706a1fbf" containerName="payload-processing" Nov 01 22:30:36.152961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.152956 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="552811b9-12a3-45e2-a681-9640706a1fbf" containerName="payload-processing" Nov 01 22:30:36.156546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.156524 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.165625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.165601 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf"] Nov 01 22:30:36.205584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.205553 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9llzd\" (UniqueName: \"kubernetes.io/projected/552811b9-12a3-45e2-a681-9640706a1fbf-kube-api-access-9llzd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:36.245996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.245922 5036 generic.go:356] "Generic (PLEG): container finished" podID="df740c66-4784-4428-8b13-90db3b380c5a" containerID="5090c6f2fdfea8850a4098f0e48f4c3a287bb670f4634d4c41be60e0eb4e45fe" exitCode=0 Nov 01 22:30:36.246133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.246013 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" event={"ID":"df740c66-4784-4428-8b13-90db3b380c5a","Type":"ContainerDied","Data":"5090c6f2fdfea8850a4098f0e48f4c3a287bb670f4634d4c41be60e0eb4e45fe"} Nov 01 22:30:36.247524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.247502 5036 generic.go:356] "Generic (PLEG): container finished" podID="552811b9-12a3-45e2-a681-9640706a1fbf" containerID="cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14" exitCode=0 Nov 01 22:30:36.247640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.247524 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" event={"ID":"552811b9-12a3-45e2-a681-9640706a1fbf","Type":"ContainerDied","Data":"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14"} Nov 01 22:30:36.247640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.247550 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" Nov 01 22:30:36.247640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.247564 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl" event={"ID":"552811b9-12a3-45e2-a681-9640706a1fbf","Type":"ContainerDied","Data":"7097ede72a9d0daf2246655ccb89cd50bd513cd0c7ef4ebf1a5e1ad9efa6e119"} Nov 01 22:30:36.247640 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.247585 5036 scope.go:122] "RemoveContainer" containerID="cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14" Nov 01 22:30:36.259532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.259513 5036 scope.go:122] "RemoveContainer" containerID="cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14" Nov 01 22:30:36.259829 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:36.259811 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14\": container with ID starting with cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14 not found: ID does not exist" containerID="cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14" Nov 01 22:30:36.259909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.259835 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14"} err="failed to get container status \"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14\": rpc error: code = NotFound desc = could not find container \"cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14\": container with ID starting with cbba08c69aa7725c6fe2aa5ce6c1da4c6948092bd5d764c46cc08f574b713f14 not found: ID does not exist" Nov 01 22:30:36.289933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.289895 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 22:30:36.297472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.295828 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-53b784-566d7f64fd-8vbvl"] Nov 01 22:30:36.307325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.307293 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.307490 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.307370 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.307620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.307593 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhvn\" (UniqueName: \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.409003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.408968 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.409184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.409046 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.409251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.409196 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhvn\" (UniqueName: \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.410343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.410313 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.411988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.411952 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.422521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.422498 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhvn\" (UniqueName: \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\") pod \"maas-api-revoke-sub-9298a90a-nwfnf\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.439688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.439657 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:36.441092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.441062 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552811b9-12a3-45e2-a681-9640706a1fbf" path="/var/lib/kubelet/pods/552811b9-12a3-45e2-a681-9640706a1fbf/volumes" Nov 01 22:30:36.443596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.443576 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:36.475606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.472778 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:36.615805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.614522 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\" (UniqueName: \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\") pod \"1a7102eb-079e-4410-96e3-8778901ce3b3\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " Nov 01 22:30:36.615805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.614722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\") pod \"1a7102eb-079e-4410-96e3-8778901ce3b3\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " Nov 01 22:30:36.615805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.614775 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\") pod \"1a7102eb-079e-4410-96e3-8778901ce3b3\" (UID: \"1a7102eb-079e-4410-96e3-8778901ce3b3\") " Nov 01 22:30:36.616864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.616805 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca" pod "1a7102eb-079e-4410-96e3-8778901ce3b3" (UID: "1a7102eb-079e-4410-96e3-8778901ce3b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:36.619355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.619325 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca" pod "1a7102eb-079e-4410-96e3-8778901ce3b3" (UID: "1a7102eb-079e-4410-96e3-8778901ce3b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:36.622851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.622822 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4" pod "1a7102eb-079e-4410-96e3-8778901ce3b3" (UID: "1a7102eb-079e-4410-96e3-8778901ce3b3"). InnerVolumeSpecName "kube-api-access-kscx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:36.716480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.716429 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscx4\" (UniqueName: \"kubernetes.io/projected/1a7102eb-079e-4410-96e3-8778901ce3b3-kube-api-access-kscx4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:36.716480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.716481 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:36.716757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.716493 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7102eb-079e-4410-96e3-8778901ce3b3-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:36.817920 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:36.817887 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3715fe7_9369_4523_bac3_fbe87c143814.slice/crio-4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695 WatchSource:0}: Error finding container 4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695: Status 404 returned error can't find the container with id 4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695 Nov 01 22:30:36.819916 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:36.819890 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf"] Nov 01 22:30:37.219523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.219485 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:37.276405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.276366 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" event={"ID":"1a7102eb-079e-4410-96e3-8778901ce3b3","Type":"ContainerDied","Data":"f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359"} Nov 01 22:30:37.276648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.276430 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6197c2d983f3bf556730e77d39f3bf429bc41ce085bd17913a036a773e86359" Nov 01 22:30:37.276648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.276553 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv" Nov 01 22:30:37.283391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.283355 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" event={"ID":"d3715fe7-9369-4523-bac3-fbe87c143814","Type":"ContainerStarted","Data":"15054c286aeddd4bdaf5d05a77bf46d21c8bf39b912bdaa94e75c7802546cdf4"} Nov 01 22:30:37.283622 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.283412 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" event={"ID":"d3715fe7-9369-4523-bac3-fbe87c143814","Type":"ContainerStarted","Data":"4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695"} Nov 01 22:30:37.307999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.307943 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" podStartSLOduration=1.307923213 podStartE2EDuration="1.307923213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:37.303111362 +0000 UTC m=+2790.996091866" watchObservedRunningTime="2026-11-01 22:30:37.307923213 +0000 UTC m=+2791.000903699" Nov 01 22:30:37.469755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.469728 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:37.630062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.629948 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\" (UniqueName: \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\") pod \"df740c66-4784-4428-8b13-90db3b380c5a\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " Nov 01 22:30:37.630062 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.630010 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\") pod \"df740c66-4784-4428-8b13-90db3b380c5a\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " Nov 01 22:30:37.630325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.630150 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\") pod \"df740c66-4784-4428-8b13-90db3b380c5a\" (UID: \"df740c66-4784-4428-8b13-90db3b380c5a\") " Nov 01 22:30:37.630428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.630400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca" pod "df740c66-4784-4428-8b13-90db3b380c5a" (UID: "df740c66-4784-4428-8b13-90db3b380c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:37.630768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.630750 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:37.632204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.632182 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl" pod "df740c66-4784-4428-8b13-90db3b380c5a" (UID: "df740c66-4784-4428-8b13-90db3b380c5a"). InnerVolumeSpecName "kube-api-access-kxpnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:37.632204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.632186 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca" pod "df740c66-4784-4428-8b13-90db3b380c5a" (UID: "df740c66-4784-4428-8b13-90db3b380c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:37.720655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.720627 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:37.732578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.732553 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpnl\" (UniqueName: \"kubernetes.io/projected/df740c66-4784-4428-8b13-90db3b380c5a-kube-api-access-kxpnl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:37.732578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.732579 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df740c66-4784-4428-8b13-90db3b380c5a-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:37.906085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:37.905998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:38.293511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.293465 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" Nov 01 22:30:38.293684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.293474 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc" event={"ID":"df740c66-4784-4428-8b13-90db3b380c5a","Type":"ContainerDied","Data":"0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890"} Nov 01 22:30:38.293684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.293625 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d2412a8231b19b5587da8eada0349691c275d62391cb278e855a6b01f011890" Nov 01 22:30:38.296344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.296319 5036 generic.go:356] "Generic (PLEG): container finished" podID="d3715fe7-9369-4523-bac3-fbe87c143814" containerID="15054c286aeddd4bdaf5d05a77bf46d21c8bf39b912bdaa94e75c7802546cdf4" exitCode=0 Nov 01 22:30:38.296633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.296614 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" event={"ID":"d3715fe7-9369-4523-bac3-fbe87c143814","Type":"ContainerDied","Data":"15054c286aeddd4bdaf5d05a77bf46d21c8bf39b912bdaa94e75c7802546cdf4"} Nov 01 22:30:38.647081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:38.646989 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:39.405884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.405806 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:39.557308 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.557280 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:39.673495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.673405 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\") pod \"d3715fe7-9369-4523-bac3-fbe87c143814\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " Nov 01 22:30:39.673495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.673479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\" (UniqueName: \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\") pod \"d3715fe7-9369-4523-bac3-fbe87c143814\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " Nov 01 22:30:39.673700 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.673609 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\") pod \"d3715fe7-9369-4523-bac3-fbe87c143814\" (UID: \"d3715fe7-9369-4523-bac3-fbe87c143814\") " Nov 01 22:30:39.673989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.673962 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca" pod "d3715fe7-9369-4523-bac3-fbe87c143814" (UID: "d3715fe7-9369-4523-bac3-fbe87c143814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:39.674129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.674111 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:39.675528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.675505 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca" pod "d3715fe7-9369-4523-bac3-fbe87c143814" (UID: "d3715fe7-9369-4523-bac3-fbe87c143814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:39.675725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.675703 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn" pod "d3715fe7-9369-4523-bac3-fbe87c143814" (UID: "d3715fe7-9369-4523-bac3-fbe87c143814"). InnerVolumeSpecName "kube-api-access-zbhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:39.775244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.775161 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3715fe7-9369-4523-bac3-fbe87c143814-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:39.775244 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:39.775189 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhvn\" (UniqueName: \"kubernetes.io/projected/d3715fe7-9369-4523-bac3-fbe87c143814-kube-api-access-zbhvn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.232229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:40.232190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.322246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:40.322210 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" Nov 01 22:30:40.322246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:40.322238 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf" event={"ID":"d3715fe7-9369-4523-bac3-fbe87c143814","Type":"ContainerDied","Data":"4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695"} Nov 01 22:30:40.322246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:40.322296 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c85a5a5ab92420d0cffaf94de3ededafc157910d3e4d86a4b613cd08b34b695" Nov 01 22:30:41.033265 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:41.033226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:41.075876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:41.075831 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:30:41.988797 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:41.985333 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:43.044190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:43.044153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:44.022938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:44.022891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:44.355113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:44.355027 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:30:44.442483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:44.439714 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:44.606518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:44.603185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:44.926639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:44.926549 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:45.567208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:45.567138 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:46.253588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:46.253551 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:46.800051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:46.799971 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:46.978508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:46.976992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:47.155636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.155497 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc"] Nov 01 22:30:47.160613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.159231 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:47.201604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.197514 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2c5135-n2gkc"] Nov 01 22:30:47.284919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.284881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:47.597865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.597835 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:47.642685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.642651 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9"] Nov 01 22:30:47.643271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643256 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1a7102eb-079e-4410-96e3-8778901ce3b3" containerName="revoke-keys" Nov 01 22:30:47.643315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643273 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7102eb-079e-4410-96e3-8778901ce3b3" containerName="revoke-keys" Nov 01 22:30:47.643315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643300 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d3715fe7-9369-4523-bac3-fbe87c143814" containerName="revoke-keys" Nov 01 22:30:47.643315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643306 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3715fe7-9369-4523-bac3-fbe87c143814" containerName="revoke-keys" Nov 01 22:30:47.643417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643322 5036 cpu_manager.go:414] "containerMap: removing container" podUID="df740c66-4784-4428-8b13-90db3b380c5a" containerName="revoke-keys" Nov 01 22:30:47.643417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643327 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df740c66-4784-4428-8b13-90db3b380c5a" containerName="revoke-keys" Nov 01 22:30:47.643505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643425 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7102eb-079e-4410-96e3-8778901ce3b3" containerName="revoke-keys" Nov 01 22:30:47.643505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643437 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3715fe7-9369-4523-bac3-fbe87c143814" containerName="revoke-keys" Nov 01 22:30:47.643505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.643458 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="df740c66-4784-4428-8b13-90db3b380c5a" containerName="revoke-keys" Nov 01 22:30:47.647339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.647321 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.651680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.651651 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:47.652639 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.652617 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:30:47.655689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.655661 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9"] Nov 01 22:30:47.720520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.720484 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:47.749311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.749274 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.749610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.749331 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.749723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.749699 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vhs\" (UniqueName: \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.850835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.850748 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vhs\" (UniqueName: \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.851374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.850848 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.851374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.850892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.851831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.851792 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.853532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.853505 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.863338 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.863302 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vhs\" (UniqueName: \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\") pod \"maas-api-revoke-sub-16d3c04f-xqvj9\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:47.966902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:47.966866 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:48.228934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:48.228851 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:48.300017 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:48.299976 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44cc163_afce_48d0_9f83_5c531b9fd46f.slice/crio-115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a WatchSource:0}: Error finding container 115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a: Status 404 returned error can't find the container with id 115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a Nov 01 22:30:48.302136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:48.302101 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9"] Nov 01 22:30:48.439370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:48.439341 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df740c66-4784-4428-8b13-90db3b380c5a" path="/var/lib/kubelet/pods/df740c66-4784-4428-8b13-90db3b380c5a/volumes" Nov 01 22:30:48.543510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:48.543478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" event={"ID":"b44cc163-afce-48d0-9f83-5c531b9fd46f","Type":"ContainerStarted","Data":"115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a"} Nov 01 22:30:48.859479 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:48.859407 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:30:48.927470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:48.918771 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:49.261587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.261553 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh"] Nov 01 22:30:49.265807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.265787 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.276407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.276379 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh"] Nov 01 22:30:49.355528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.355483 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" containerID="cri-o://114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c" gracePeriod=25 Nov 01 22:30:49.379795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.379757 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd64l\" (UniqueName: \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.379993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.379972 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.380057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.380042 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.481237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.481153 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.481237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.481236 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.481527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.481317 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd64l\" (UniqueName: \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.481946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.481905 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.484070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.484042 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.494361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.494335 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd64l\" (UniqueName: \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\") pod \"maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.555298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.555265 5036 generic.go:356] "Generic (PLEG): container finished" podID="b44cc163-afce-48d0-9f83-5c531b9fd46f" containerID="8eadd75bd19dd7399a899a475e3d12c06a0454986882df0c4a8de0b26f9b8b52" exitCode=0 Nov 01 22:30:49.555466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.555348 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" event={"ID":"b44cc163-afce-48d0-9f83-5c531b9fd46f","Type":"ContainerDied","Data":"8eadd75bd19dd7399a899a475e3d12c06a0454986882df0c4a8de0b26f9b8b52"} Nov 01 22:30:49.587952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.587921 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:49.883481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.883086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:49.906807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:49.906771 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh"] Nov 01 22:30:49.907614 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:30:49.907581 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553d618b_48e9_492f_bebb_f848a7ec4f17.slice/crio-2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3 WatchSource:0}: Error finding container 2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3: Status 404 returned error can't find the container with id 2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3 Nov 01 22:30:50.441080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.441040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:50.577854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.577760 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" event={"ID":"553d618b-48e9-492f-bebb-f848a7ec4f17","Type":"ContainerStarted","Data":"dc357fc0fd6ad371146be8584ccd17380c31b85bd875f6034db3eef087c21ecf"} Nov 01 22:30:50.578134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.578097 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" event={"ID":"553d618b-48e9-492f-bebb-f848a7ec4f17","Type":"ContainerStarted","Data":"2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3"} Nov 01 22:30:50.580081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.580056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:50.699469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.696146 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" podStartSLOduration=1.6961230600000001 podStartE2EDuration="1.69612306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:50.627056727 +0000 UTC m=+2804.320037214" watchObservedRunningTime="2026-11-01 22:30:50.69612306 +0000 UTC m=+2804.389103547" Nov 01 22:30:50.855972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.854420 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:50.890280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.884274 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv"] Nov 01 22:30:50.891500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.891468 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6bb56-hkmjv"] Nov 01 22:30:50.990536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:50.990499 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.045034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.045003 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.058273 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\") pod \"b44cc163-afce-48d0-9f83-5c531b9fd46f\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.058345 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\" (UniqueName: \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\") pod \"b44cc163-afce-48d0-9f83-5c531b9fd46f\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.058561 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\") pod \"b44cc163-afce-48d0-9f83-5c531b9fd46f\" (UID: \"b44cc163-afce-48d0-9f83-5c531b9fd46f\") " Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.061930 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca" pod "b44cc163-afce-48d0-9f83-5c531b9fd46f" (UID: "b44cc163-afce-48d0-9f83-5c531b9fd46f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.066352 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs" pod "b44cc163-afce-48d0-9f83-5c531b9fd46f" (UID: "b44cc163-afce-48d0-9f83-5c531b9fd46f"). InnerVolumeSpecName "kube-api-access-r7vhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:51.103467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.067671 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca" pod "b44cc163-afce-48d0-9f83-5c531b9fd46f" (UID: "b44cc163-afce-48d0-9f83-5c531b9fd46f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:51.135105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.135005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.160236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.160207 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.160389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.160237 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vhs\" (UniqueName: \"kubernetes.io/projected/b44cc163-afce-48d0-9f83-5c531b9fd46f-kube-api-access-r7vhs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.160389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.160256 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cc163-afce-48d0-9f83-5c531b9fd46f-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.372236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.372191 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.589072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.589039 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" Nov 01 22:30:51.589495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.589091 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9" event={"ID":"b44cc163-afce-48d0-9f83-5c531b9fd46f","Type":"ContainerDied","Data":"115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a"} Nov 01 22:30:51.589495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.589328 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="115585b8c33fb18669693c7a5c0a4ad2b5544bb63f3014fce0bcffe3946e891a" Nov 01 22:30:51.591233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.591208 5036 generic.go:356] "Generic (PLEG): container finished" podID="553d618b-48e9-492f-bebb-f848a7ec4f17" containerID="dc357fc0fd6ad371146be8584ccd17380c31b85bd875f6034db3eef087c21ecf" exitCode=0 Nov 01 22:30:51.591385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.591291 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" event={"ID":"553d618b-48e9-492f-bebb-f848a7ec4f17","Type":"ContainerDied","Data":"dc357fc0fd6ad371146be8584ccd17380c31b85bd875f6034db3eef087c21ecf"} Nov 01 22:30:51.934892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:51.934785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:52.445805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.445764 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7102eb-079e-4410-96e3-8778901ce3b3" path="/var/lib/kubelet/pods/1a7102eb-079e-4410-96e3-8778901ce3b3/volumes" Nov 01 22:30:52.521375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.521340 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:52.669467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.668944 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:52.899480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.899032 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:52.970964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.970435 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5"] Nov 01 22:30:52.974925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:52.974528 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-53b784-xkhm5"] Nov 01 22:30:53.034021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.031880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:53.049870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.049844 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:53.163285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.163204 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:53.173837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.173809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\" (UniqueName: \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\") pod \"553d618b-48e9-492f-bebb-f848a7ec4f17\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " Nov 01 22:30:53.173981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.173892 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\") pod \"553d618b-48e9-492f-bebb-f848a7ec4f17\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " Nov 01 22:30:53.173981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.173923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\") pod \"553d618b-48e9-492f-bebb-f848a7ec4f17\" (UID: \"553d618b-48e9-492f-bebb-f848a7ec4f17\") " Nov 01 22:30:53.174318 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.174291 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca" pod "553d618b-48e9-492f-bebb-f848a7ec4f17" (UID: "553d618b-48e9-492f-bebb-f848a7ec4f17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:53.174488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.174468 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:53.176146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.176122 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l" pod "553d618b-48e9-492f-bebb-f848a7ec4f17" (UID: "553d618b-48e9-492f-bebb-f848a7ec4f17"). InnerVolumeSpecName "kube-api-access-fd64l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:53.176146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.176119 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca" pod "553d618b-48e9-492f-bebb-f848a7ec4f17" (UID: "553d618b-48e9-492f-bebb-f848a7ec4f17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:53.275568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.275537 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd64l\" (UniqueName: \"kubernetes.io/projected/553d618b-48e9-492f-bebb-f848a7ec4f17-kube-api-access-fd64l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:53.275568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.275569 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553d618b-48e9-492f-bebb-f848a7ec4f17-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:53.307612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.307572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:53.619927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.619894 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" Nov 01 22:30:53.620108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.619901 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh" event={"ID":"553d618b-48e9-492f-bebb-f848a7ec4f17","Type":"ContainerDied","Data":"2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3"} Nov 01 22:30:53.620108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.620030 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aabe1e9d0f2e4a27ec1ec0681ba2a764d6d8e8018403ee5d118300e0c711ee3" Nov 01 22:30:53.961895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:53.961797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:54.360956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:54.360912 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:54.440329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:54.440296 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554bc699-0b01-4212-92ae-5716cb2ad921" path="/var/lib/kubelet/pods/554bc699-0b01-4212-92ae-5716cb2ad921/volumes" Nov 01 22:30:54.494958 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:54.494921 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:54.666834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:54.666748 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:55.045078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:55.040284 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:55.178908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:55.178869 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:55.986057 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:55.985972 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:56.037476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.037434 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 22:30:56.051685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.051661 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\" (UniqueName: \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\") pod \"32ba525e-35ac-421c-9f61-4de3d469c3a0\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " Nov 01 22:30:56.051831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.051813 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\") pod \"32ba525e-35ac-421c-9f61-4de3d469c3a0\" (UID: \"32ba525e-35ac-421c-9f61-4de3d469c3a0\") " Nov 01 22:30:56.052162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.052140 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume" pod "32ba525e-35ac-421c-9f61-4de3d469c3a0" (UID: "32ba525e-35ac-421c-9f61-4de3d469c3a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:56.052358 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.052333 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ba525e-35ac-421c-9f61-4de3d469c3a0-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:56.053952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.053926 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf" pod "32ba525e-35ac-421c-9f61-4de3d469c3a0" (UID: "32ba525e-35ac-421c-9f61-4de3d469c3a0"). InnerVolumeSpecName "kube-api-access-q2hnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:56.153166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.153122 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hnf\" (UniqueName: \"kubernetes.io/projected/32ba525e-35ac-421c-9f61-4de3d469c3a0-kube-api-access-q2hnf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:56.196124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.196077 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:56.659147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.659112 5036 generic.go:356] "Generic (PLEG): container finished" podID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerID="86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629" exitCode=0 Nov 01 22:30:56.659301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.659163 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" Nov 01 22:30:56.659301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.659195 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" event={"ID":"32ba525e-35ac-421c-9f61-4de3d469c3a0","Type":"ContainerDied","Data":"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629"} Nov 01 22:30:56.659301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.659249 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb" event={"ID":"32ba525e-35ac-421c-9f61-4de3d469c3a0","Type":"ContainerDied","Data":"8f3c6cdefeaef82a9e43c0597d18f0c5c7adf122e2aa051642c04f060782ddb4"} Nov 01 22:30:56.659301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.659278 5036 scope.go:122] "RemoveContainer" containerID="86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629" Nov 01 22:30:56.670549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.670531 5036 scope.go:122] "RemoveContainer" containerID="86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629" Nov 01 22:30:56.670831 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:30:56.670807 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629\": container with ID starting with 86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629 not found: ID does not exist" containerID="86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629" Nov 01 22:30:56.670879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.670840 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629"} err="failed to get container status \"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629\": rpc error: code = NotFound desc = could not find container \"86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629\": container with ID starting with 86341ce0ad14cc2bb300eea70bf6db3151f2cda51a48efee9d5400fbbf4b0629 not found: ID does not exist" Nov 01 22:30:56.678982 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.678958 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 22:30:56.685282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:56.685257 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-53b784-5db67d6c99-vzhvb"] Nov 01 22:30:57.026075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:57.026044 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:57.745134 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:57.745099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:57.874467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:57.874265 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:58.044929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:58.044890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:58.439669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:58.439586 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" path="/var/lib/kubelet/pods/32ba525e-35ac-421c-9f61-4de3d469c3a0/volumes" Nov 01 22:30:58.824348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:58.824211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:59.804477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:30:59.803730 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:00.031730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.031669 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb"] Nov 01 22:31:00.038780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.038743 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb860254-4hmqb"] Nov 01 22:31:00.044787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.044726 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4"] Nov 01 22:31:00.048314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.048293 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2096d7c-4bzd4"] Nov 01 22:31:00.056706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.056645 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 22:31:00.069180 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.069155 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 22:31:00.211752 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.211712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:00.443560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.443463 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9571de-d00a-4849-a3f9-e2b7feebd067" path="/var/lib/kubelet/pods/8e9571de-d00a-4849-a3f9-e2b7feebd067/volumes" Nov 01 22:31:00.444156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.444132 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84f35aa-3b65-4a84-a0a8-0d33ff418a1f" path="/var/lib/kubelet/pods/f84f35aa-3b65-4a84-a0a8-0d33ff418a1f/volumes" Nov 01 22:31:00.824558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:00.824526 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:01.158473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.158372 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:01.299078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.299040 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:01.526234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.526192 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 22:31:01.526986 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.526907 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" containerID="cri-o://2d09c076f2867e094975cd3b50ba04e35972a9cd0ca1fad5584dccc31c61608f" gracePeriod=30 Nov 01 22:31:01.563185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.563153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:01.810517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:01.805743 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:02.320943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:02.320902 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:02.452978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:02.452942 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:02.583125 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:02.583035 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:03.410470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:03.398998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:04.083644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:04.083604 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:04.339147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:04.339059 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:05.230121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.230084 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:05.354541 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.354496 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 22:31:05.368223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.368190 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 22:31:05.460263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.460224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:05.592041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.591990 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:05.959880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.959793 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d"] Nov 01 22:31:05.960577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960552 5036 cpu_manager.go:414] "containerMap: removing container" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" Nov 01 22:31:05.960577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960576 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" Nov 01 22:31:05.960731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960590 5036 cpu_manager.go:414] "containerMap: removing container" podUID="b44cc163-afce-48d0-9f83-5c531b9fd46f" containerName="revoke-keys" Nov 01 22:31:05.960731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960596 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44cc163-afce-48d0-9f83-5c531b9fd46f" containerName="revoke-keys" Nov 01 22:31:05.960731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960630 5036 cpu_manager.go:414] "containerMap: removing container" podUID="553d618b-48e9-492f-bebb-f848a7ec4f17" containerName="revoke-keys" Nov 01 22:31:05.960731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960637 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553d618b-48e9-492f-bebb-f848a7ec4f17" containerName="revoke-keys" Nov 01 22:31:05.960904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960831 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="553d618b-48e9-492f-bebb-f848a7ec4f17" containerName="revoke-keys" Nov 01 22:31:05.960904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960853 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ba525e-35ac-421c-9f61-4de3d469c3a0" containerName="payload-pre-processing" Nov 01 22:31:05.960904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.960861 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44cc163-afce-48d0-9f83-5c531b9fd46f" containerName="revoke-keys" Nov 01 22:31:05.964135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.964101 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:05.969077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.969051 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:05.969949 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.969929 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:05.979428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.979404 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d"] Nov 01 22:31:05.991928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:05.991888 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:06.070905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.070872 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.071109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.071088 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.071195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.071181 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27sts\" (UniqueName: \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.172345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.172309 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sts\" (UniqueName: \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.172567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.172410 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.172672 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.172651 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.173205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.173184 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.175241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.175219 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.184480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.184437 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27sts\" (UniqueName: \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\") pod \"maas-api-revoke-sub-4f886d75-mww6d\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.230921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.230858 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:06.297747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.297712 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:06.588789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.588750 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d"] Nov 01 22:31:06.613766 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:31:06.613737 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422f341f_492d_476d_b15d_5d3c9654c7d5.slice/crio-0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9 WatchSource:0}: Error finding container 0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9: Status 404 returned error can't find the container with id 0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9 Nov 01 22:31:06.770564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.770486 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" event={"ID":"422f341f-492d-476d-b15d-5d3c9654c7d5","Type":"ContainerStarted","Data":"0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9"} Nov 01 22:31:06.774400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.774368 5036 generic.go:356] "Generic (PLEG): container finished" podID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerID="2d09c076f2867e094975cd3b50ba04e35972a9cd0ca1fad5584dccc31c61608f" exitCode=0 Nov 01 22:31:06.774588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.774425 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" event={"ID":"cfb90432-17a2-41b1-9fbc-926a535dfc14","Type":"ContainerDied","Data":"2d09c076f2867e094975cd3b50ba04e35972a9cd0ca1fad5584dccc31c61608f"} Nov 01 22:31:06.803334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.803275 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884115 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884216 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884314 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884411 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884712 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884770 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884838 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884923 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884945 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.884989 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\") pod \"cfb90432-17a2-41b1-9fbc-926a535dfc14\" (UID: \"cfb90432-17a2-41b1-9fbc-926a535dfc14\") " Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.885206 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.885696 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb90432-17a2-41b1-9fbc-926a535dfc14-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.885720 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.886031 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.886177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.886234 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.888289 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "kube-api-access-vr8tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.888326 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.888350 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:06.890497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.889949 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy" pod "cfb90432-17a2-41b1-9fbc-926a535dfc14" (UID: "cfb90432-17a2-41b1-9fbc-926a535dfc14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:06.921291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.921260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:06.988367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988329 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988361 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988377 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988392 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988422 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8tp\" (UniqueName: \"kubernetes.io/projected/cfb90432-17a2-41b1-9fbc-926a535dfc14-kube-api-access-vr8tp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988438 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:06.988690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:06.988468 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb90432-17a2-41b1-9fbc-926a535dfc14-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:07.658188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.658149 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:07.786143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.786109 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" Nov 01 22:31:07.786143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.786127 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw" event={"ID":"cfb90432-17a2-41b1-9fbc-926a535dfc14","Type":"ContainerDied","Data":"5a3637f7a246b882c3b80cdd99be0346023e29a4b770e6ff2f8d4b9f9cce62bb"} Nov 01 22:31:07.786398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.786208 5036 scope.go:122] "RemoveContainer" containerID="2d09c076f2867e094975cd3b50ba04e35972a9cd0ca1fad5584dccc31c61608f" Nov 01 22:31:07.788754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.788723 5036 generic.go:356] "Generic (PLEG): container finished" podID="422f341f-492d-476d-b15d-5d3c9654c7d5" containerID="996ecc14d4b1a4a2fcda877aa454cd0d964be4b4293cb16640fd27412ed58096" exitCode=0 Nov 01 22:31:07.788927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.788817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" event={"ID":"422f341f-492d-476d-b15d-5d3c9654c7d5","Type":"ContainerDied","Data":"996ecc14d4b1a4a2fcda877aa454cd0d964be4b4293cb16640fd27412ed58096"} Nov 01 22:31:07.832357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.832332 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 22:31:07.836187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:07.836164 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-zk4fw"] Nov 01 22:31:08.440168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:08.440132 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" path="/var/lib/kubelet/pods/cfb90432-17a2-41b1-9fbc-926a535dfc14/volumes" Nov 01 22:31:08.460912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:08.460877 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:08.982872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:08.982842 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:09.027171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.027139 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s"] Nov 01 22:31:09.031224 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.031187 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948e632f-db27s"] Nov 01 22:31:09.132283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.132203 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\" (UniqueName: \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\") pod \"422f341f-492d-476d-b15d-5d3c9654c7d5\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " Nov 01 22:31:09.132283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.132275 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\") pod \"422f341f-492d-476d-b15d-5d3c9654c7d5\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " Nov 01 22:31:09.132562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.132381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\") pod \"422f341f-492d-476d-b15d-5d3c9654c7d5\" (UID: \"422f341f-492d-476d-b15d-5d3c9654c7d5\") " Nov 01 22:31:09.132763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.132735 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca" pod "422f341f-492d-476d-b15d-5d3c9654c7d5" (UID: "422f341f-492d-476d-b15d-5d3c9654c7d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:09.134547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.134521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca" pod "422f341f-492d-476d-b15d-5d3c9654c7d5" (UID: "422f341f-492d-476d-b15d-5d3c9654c7d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:09.134681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.134551 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts" pod "422f341f-492d-476d-b15d-5d3c9654c7d5" (UID: "422f341f-492d-476d-b15d-5d3c9654c7d5"). InnerVolumeSpecName "kube-api-access-27sts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:09.233794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.233751 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27sts\" (UniqueName: \"kubernetes.io/projected/422f341f-492d-476d-b15d-5d3c9654c7d5-kube-api-access-27sts\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.233794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.233784 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.233794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.233795 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f341f-492d-476d-b15d-5d3c9654c7d5-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.246084 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.246042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:09.648220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.648189 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:31:09.746037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.745955 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\") pod \"01302463-9a3a-46cf-a162-d83e9a846249\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " Nov 01 22:31:09.746195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.746060 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\" (UniqueName: \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\") pod \"01302463-9a3a-46cf-a162-d83e9a846249\" (UID: \"01302463-9a3a-46cf-a162-d83e9a846249\") " Nov 01 22:31:09.746342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.746317 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume" pod "01302463-9a3a-46cf-a162-d83e9a846249" (UID: "01302463-9a3a-46cf-a162-d83e9a846249"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:09.746559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.746538 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01302463-9a3a-46cf-a162-d83e9a846249-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.748195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.748172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc" pod "01302463-9a3a-46cf-a162-d83e9a846249" (UID: "01302463-9a3a-46cf-a162-d83e9a846249"). InnerVolumeSpecName "kube-api-access-nl6dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:09.813801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.813770 5036 generic.go:356] "Generic (PLEG): container finished" podID="01302463-9a3a-46cf-a162-d83e9a846249" containerID="114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c" exitCode=0 Nov 01 22:31:09.813950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.813814 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" Nov 01 22:31:09.813950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.813857 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" event={"ID":"01302463-9a3a-46cf-a162-d83e9a846249","Type":"ContainerDied","Data":"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c"} Nov 01 22:31:09.813950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.813918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67" event={"ID":"01302463-9a3a-46cf-a162-d83e9a846249","Type":"ContainerDied","Data":"eabc2de46bd40305a4d0ffd459ab32c160ba82f44af2b0d742650d3893e6a718"} Nov 01 22:31:09.814098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.813949 5036 scope.go:122] "RemoveContainer" containerID="114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c" Nov 01 22:31:09.815926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.815896 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" Nov 01 22:31:09.816063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.815920 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d" event={"ID":"422f341f-492d-476d-b15d-5d3c9654c7d5","Type":"ContainerDied","Data":"0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9"} Nov 01 22:31:09.816063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.815966 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b29f89b568852bd00a2e69497d22636f8f43102e63d3eb3c47f452e7bfef4f9" Nov 01 22:31:09.828767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.828748 5036 scope.go:122] "RemoveContainer" containerID="114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c" Nov 01 22:31:09.829039 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:09.829020 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c\": container with ID starting with 114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c not found: ID does not exist" containerID="114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c" Nov 01 22:31:09.829102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.829049 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c"} err="failed to get container status \"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c\": rpc error: code = NotFound desc = could not find container \"114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c\": container with ID starting with 114e2e37c2d57536a838880bc53f734700be1f90aaac889e15f57c7300495c9c not found: ID does not exist" Nov 01 22:31:09.841514 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.841490 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:31:09.846000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.845979 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2c5135-66556bcd99-7fs67"] Nov 01 22:31:09.847327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:09.847303 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl6dc\" (UniqueName: \"kubernetes.io/projected/01302463-9a3a-46cf-a162-d83e9a846249-kube-api-access-nl6dc\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:10.050349 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.050295 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:31:10.106330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.106287 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:10.255467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.252069 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:10.355135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.355033 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" podUID="a6ad7920-1e49-4002-9ca8-f717e36114bb" containerName="payload-processing" containerID="cri-o://0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca" gracePeriod=25 Nov 01 22:31:10.368924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.368890 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" containerID="cri-o://ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f" gracePeriod=25 Nov 01 22:31:10.440187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.440151 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01302463-9a3a-46cf-a162-d83e9a846249" path="/var/lib/kubelet/pods/01302463-9a3a-46cf-a162-d83e9a846249/volumes" Nov 01 22:31:10.440721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.440704 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb1ca90-5597-4567-b724-f5e7e5a5cf33" path="/var/lib/kubelet/pods/7cb1ca90-5597-4567-b724-f5e7e5a5cf33/volumes" Nov 01 22:31:10.703953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.703920 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 22:31:10.831204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.831168 5036 generic.go:356] "Generic (PLEG): container finished" podID="a6ad7920-1e49-4002-9ca8-f717e36114bb" containerID="0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca" exitCode=0 Nov 01 22:31:10.831390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.831209 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" Nov 01 22:31:10.831390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.831249 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" event={"ID":"a6ad7920-1e49-4002-9ca8-f717e36114bb","Type":"ContainerDied","Data":"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca"} Nov 01 22:31:10.831390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.831298 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6" event={"ID":"a6ad7920-1e49-4002-9ca8-f717e36114bb","Type":"ContainerDied","Data":"66b36ccc60defd505dbd04d31488a9de67f2adad16a9f4fa7907747d817c1759"} Nov 01 22:31:10.831390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.831319 5036 scope.go:122] "RemoveContainer" containerID="0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca" Nov 01 22:31:10.843250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.843229 5036 scope.go:122] "RemoveContainer" containerID="0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca" Nov 01 22:31:10.843576 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:10.843552 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca\": container with ID starting with 0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca not found: ID does not exist" containerID="0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca" Nov 01 22:31:10.843717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.843580 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca"} err="failed to get container status \"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca\": rpc error: code = NotFound desc = could not find container \"0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca\": container with ID starting with 0cfb658f9b64e91c2bce584ae54d0ee15eec37e7b2447c66dfa95183b793a6ca not found: ID does not exist" Nov 01 22:31:10.871119 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.871098 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\" (UniqueName: \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\") pod \"a6ad7920-1e49-4002-9ca8-f717e36114bb\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " Nov 01 22:31:10.871254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.871239 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\") pod \"a6ad7920-1e49-4002-9ca8-f717e36114bb\" (UID: \"a6ad7920-1e49-4002-9ca8-f717e36114bb\") " Nov 01 22:31:10.871662 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.871632 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume" pod "a6ad7920-1e49-4002-9ca8-f717e36114bb" (UID: "a6ad7920-1e49-4002-9ca8-f717e36114bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:10.873195 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.873172 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z" pod "a6ad7920-1e49-4002-9ca8-f717e36114bb" (UID: "a6ad7920-1e49-4002-9ca8-f717e36114bb"). InnerVolumeSpecName "kube-api-access-pz54z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:10.879747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.879704 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:10.991530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.990658 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz54z\" (UniqueName: \"kubernetes.io/projected/a6ad7920-1e49-4002-9ca8-f717e36114bb-kube-api-access-pz54z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:10.991530 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:10.990701 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ad7920-1e49-4002-9ca8-f717e36114bb-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:11.011172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:11.011140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:11.156685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:11.156649 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 22:31:11.161970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:11.161942 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b0b7a3-769756b4c4-gcwr6"] Nov 01 22:31:11.733462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:11.733409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:12.441741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:12.441703 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ad7920-1e49-4002-9ca8-f717e36114bb" path="/var/lib/kubelet/pods/a6ad7920-1e49-4002-9ca8-f717e36114bb/volumes" Nov 01 22:31:12.669147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:12.669107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:13.523681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:13.521258 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:13.648388 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:13.648352 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:13.791527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:13.790970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:14.014199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:14.014144 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:14.186136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:14.186052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:14.330293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:14.330239 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:31:14.386535 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:14.386407 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:14.545511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:14.545430 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:15.084107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.084069 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" containerID="cri-o://d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424" gracePeriod=45 Nov 01 22:31:15.095887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.095852 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" containerID="cri-o://4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89" gracePeriod=45 Nov 01 22:31:15.398658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.398607 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:31:15.428368 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.428341 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:31:15.498327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498287 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\" (UniqueName: \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.498563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498391 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.498654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.498654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498632 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.498769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498680 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\" (UniqueName: \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.498834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498790 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.498908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498886 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.498975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498928 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\" (UniqueName: \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.499040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.498984 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.499040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.499013 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\") pod \"9aabc696-b098-4ffa-b640-7fbf8eddf008\" (UID: \"9aabc696-b098-4ffa-b640-7fbf8eddf008\") " Nov 01 22:31:15.499146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.499049 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.499146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.499086 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\" (UniqueName: \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\") pod \"e625cc85-0d4e-4677-a3ab-c05e6291d719\" (UID: \"e625cc85-0d4e-4677-a3ab-c05e6291d719\") " Nov 01 22:31:15.500065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.500033 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.501031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.501004 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.501746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.501711 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.502004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.501925 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.502187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.502162 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.502288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.502183 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.504272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504226 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504261 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504277 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504290 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504303 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504315 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.504835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.504516 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:15.506104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.506074 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:15.509779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.509571 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.509779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.509735 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "kube-api-access-477q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:15.510966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.510931 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm" pod "9aabc696-b098-4ffa-b640-7fbf8eddf008" (UID: "9aabc696-b098-4ffa-b640-7fbf8eddf008"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:15.512702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.512029 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z" pod "e625cc85-0d4e-4677-a3ab-c05e6291d719" (UID: "e625cc85-0d4e-4677-a3ab-c05e6291d719"). InnerVolumeSpecName "kube-api-access-gd96z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:15.554117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.554015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:15.605363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605328 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aabc696-b098-4ffa-b640-7fbf8eddf008-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.605363 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605363 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-477q7\" (UniqueName: \"kubernetes.io/projected/9aabc696-b098-4ffa-b640-7fbf8eddf008-kube-api-access-477q7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.605623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605374 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd96z\" (UniqueName: \"kubernetes.io/projected/e625cc85-0d4e-4677-a3ab-c05e6291d719-kube-api-access-gd96z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.605623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605387 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e625cc85-0d4e-4677-a3ab-c05e6291d719-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.605623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605402 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aabc696-b098-4ffa-b640-7fbf8eddf008-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.605623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.605412 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e625cc85-0d4e-4677-a3ab-c05e6291d719-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:15.888431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.888393 5036 generic.go:356] "Generic (PLEG): container finished" podID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerID="4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89" exitCode=0 Nov 01 22:31:15.888431 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.888434 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" Nov 01 22:31:15.888736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.888478 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" event={"ID":"9aabc696-b098-4ffa-b640-7fbf8eddf008","Type":"ContainerDied","Data":"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89"} Nov 01 22:31:15.888736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.888527 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g" event={"ID":"9aabc696-b098-4ffa-b640-7fbf8eddf008","Type":"ContainerDied","Data":"6416a44f47adc72f071175a629dd6fadab2f63817ddfe0f97b794c567bea9938"} Nov 01 22:31:15.888736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.888552 5036 scope.go:122] "RemoveContainer" containerID="4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89" Nov 01 22:31:15.891098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.890903 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" Nov 01 22:31:15.891253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.891170 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" event={"ID":"e625cc85-0d4e-4677-a3ab-c05e6291d719","Type":"ContainerDied","Data":"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424"} Nov 01 22:31:15.898531 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.898498 5036 generic.go:356] "Generic (PLEG): container finished" podID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerID="d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424" exitCode=0 Nov 01 22:31:15.898679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.898553 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz" event={"ID":"e625cc85-0d4e-4677-a3ab-c05e6291d719","Type":"ContainerDied","Data":"d3a3e5c6b19dcbb0e25235b5ac4e47fb03d8bd1cff7e151254a318b87c84a6e9"} Nov 01 22:31:15.902645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.902575 5036 scope.go:122] "RemoveContainer" containerID="4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89" Nov 01 22:31:15.902935 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:15.902909 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89\": container with ID starting with 4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89 not found: ID does not exist" containerID="4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89" Nov 01 22:31:15.903023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.902946 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89"} err="failed to get container status \"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89\": rpc error: code = NotFound desc = could not find container \"4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89\": container with ID starting with 4880f6be6aa319efd72df06b6db99453692341acbd23ef2d183dcee34a4cca89 not found: ID does not exist" Nov 01 22:31:15.903023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.902969 5036 scope.go:122] "RemoveContainer" containerID="d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424" Nov 01 22:31:15.913595 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.913574 5036 scope.go:122] "RemoveContainer" containerID="d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424" Nov 01 22:31:15.913870 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:15.913849 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424\": container with ID starting with d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424 not found: ID does not exist" containerID="d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424" Nov 01 22:31:15.913945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.913876 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424"} err="failed to get container status \"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424\": rpc error: code = NotFound desc = could not find container \"d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424\": container with ID starting with d34adec8347f34f47f9a4b240e38aacae204b09ef0becb2ee404eadb61856424 not found: ID does not exist" Nov 01 22:31:15.920522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.920497 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 22:31:15.924691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.924666 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/premium-w5-b0b7a3-kserve-78f7bf89fd-k8c8g"] Nov 01 22:31:15.935990 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.935968 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 22:31:15.941190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:15.941168 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-2-w5-b0b7a3-kserve-79958d8bbb-777gz"] Nov 01 22:31:16.325689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:16.325654 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:16.440174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:16.440131 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" path="/var/lib/kubelet/pods/9aabc696-b098-4ffa-b640-7fbf8eddf008/volumes" Nov 01 22:31:16.440660 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:16.440642 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" path="/var/lib/kubelet/pods/e625cc85-0d4e-4677-a3ab-c05e6291d719/volumes" Nov 01 22:31:17.020829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:17.020790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:17.671306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:17.671272 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:17.801721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:17.801681 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:17.961623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:17.961523 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh"] Nov 01 22:31:17.977183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:17.973199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:18.081467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.077635 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-09656e-jrcgh"] Nov 01 22:31:18.188461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.188094 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:18.416658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.416625 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:18.439048 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.439008 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553d618b-48e9-492f-bebb-f848a7ec4f17" path="/var/lib/kubelet/pods/553d618b-48e9-492f-bebb-f848a7ec4f17/volumes" Nov 01 22:31:18.718611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.718517 5036 scope.go:122] "RemoveContainer" containerID="b84f9394aaaafbcd3c35e6acc31015a01fea214be6c5d180bd5b15f879f21bcd" Nov 01 22:31:18.726236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:18.726200 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:18.862534 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:18.862494 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:31:19.026680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:19.026649 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:19.928055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:19.928020 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:20.600644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:20.600610 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:20.754940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:20.754886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:20.900849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:20.900747 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:21.043194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:21.043154 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:21.169218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:21.169092 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:21.413956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:21.413915 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:21.556466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:21.555987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:21.998688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:21.998581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:23.191509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:23.191418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:23.344355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:23.344322 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:31:23.445476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:23.445350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:23.840299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:23.840249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:24.528398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:24.528339 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:24.814511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:24.814409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:24.984621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:24.984584 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:25.111991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.111901 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" containerID="cri-o://cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9" gracePeriod=45 Nov 01 22:31:25.340634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.340522 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:25.450390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.450359 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:31:25.505101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505063 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.505425 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505398 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:25.505623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505596 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505822 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505893 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\" (UniqueName: \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505928 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\") pod \"dd82296e-8781-49de-8ff3-417e954b022d\" (UID: \"dd82296e-8781-49de-8ff3-417e954b022d\") " Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.506458 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.505922 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:25.507019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.506985 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:25.508571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.508534 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:25.508787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.508762 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "kube-api-access-wxtvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:25.509594 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.509570 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm" pod "dd82296e-8781-49de-8ff3-417e954b022d" (UID: "dd82296e-8781-49de-8ff3-417e954b022d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:25.607848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.607811 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.607848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.607840 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.607848 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.607852 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd82296e-8781-49de-8ff3-417e954b022d-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.608399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.607861 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxtvs\" (UniqueName: \"kubernetes.io/projected/dd82296e-8781-49de-8ff3-417e954b022d-kube-api-access-wxtvs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.608399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.607872 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd82296e-8781-49de-8ff3-417e954b022d-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.933887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.933836 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:31:25.934131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:25.934097 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" podUID="7b74dc93-39fa-4e73-97d5-3996c7da3273" containerName="maas-api" containerID="cri-o://7320cf6acc35d17dab50635b891bce18237ea2a80f1fad1efab9b133f79c2f23" gracePeriod=30 Nov 01 22:31:26.096409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.096313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:26.182098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.181917 5036 generic.go:356] "Generic (PLEG): container finished" podID="7b74dc93-39fa-4e73-97d5-3996c7da3273" containerID="7320cf6acc35d17dab50635b891bce18237ea2a80f1fad1efab9b133f79c2f23" exitCode=0 Nov 01 22:31:26.182098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.182011 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" event={"ID":"7b74dc93-39fa-4e73-97d5-3996c7da3273","Type":"ContainerDied","Data":"7320cf6acc35d17dab50635b891bce18237ea2a80f1fad1efab9b133f79c2f23"} Nov 01 22:31:26.184723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.183880 5036 generic.go:356] "Generic (PLEG): container finished" podID="dd82296e-8781-49de-8ff3-417e954b022d" containerID="cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9" exitCode=0 Nov 01 22:31:26.184723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.184177 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" event={"ID":"dd82296e-8781-49de-8ff3-417e954b022d","Type":"ContainerDied","Data":"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9"} Nov 01 22:31:26.184723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.184210 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" event={"ID":"dd82296e-8781-49de-8ff3-417e954b022d","Type":"ContainerDied","Data":"bf59a31c79c19e2052e70c4c96b620ebfbb8036d9e3adf0245c213194c0fa0d4"} Nov 01 22:31:26.184723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.184233 5036 scope.go:122] "RemoveContainer" containerID="cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9" Nov 01 22:31:26.184723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.184421 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp" Nov 01 22:31:26.202889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.202684 5036 scope.go:122] "RemoveContainer" containerID="cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9" Nov 01 22:31:26.203303 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:26.203271 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9\": container with ID starting with cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9 not found: ID does not exist" containerID="cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9" Nov 01 22:31:26.203424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.203313 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9"} err="failed to get container status \"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9\": rpc error: code = NotFound desc = could not find container \"cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9\": container with ID starting with cb880f04c5307ef238bb4210ec2bb5bada8329612134ae7c8363d7b9a55451b9 not found: ID does not exist" Nov 01 22:31:26.230901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.230781 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:31:26.350474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.342006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:26.352500 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.352469 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:31:26.353173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.353140 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-521734/rate-test-model-521734-kserve-84d5785cb5-w5sqp"] Nov 01 22:31:26.439601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.439516 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd82296e-8781-49de-8ff3-417e954b022d" path="/var/lib/kubelet/pods/dd82296e-8781-49de-8ff3-417e954b022d/volumes" Nov 01 22:31:26.467260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.467230 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:26.548734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.548683 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\") pod \"7b74dc93-39fa-4e73-97d5-3996c7da3273\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " Nov 01 22:31:26.549104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.548855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\") pod \"7b74dc93-39fa-4e73-97d5-3996c7da3273\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " Nov 01 22:31:26.549104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.548936 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") pod \"7b74dc93-39fa-4e73-97d5-3996c7da3273\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " Nov 01 22:31:26.549104 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.549006 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\" (UniqueName: \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\") pod \"7b74dc93-39fa-4e73-97d5-3996c7da3273\" (UID: \"7b74dc93-39fa-4e73-97d5-3996c7da3273\") " Nov 01 22:31:26.557031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.556715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca" pod "7b74dc93-39fa-4e73-97d5-3996c7da3273" (UID: "7b74dc93-39fa-4e73-97d5-3996c7da3273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:26.560912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.560007 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls" pod "7b74dc93-39fa-4e73-97d5-3996c7da3273" (UID: "7b74dc93-39fa-4e73-97d5-3996c7da3273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:26.560912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.560150 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl" pod "7b74dc93-39fa-4e73-97d5-3996c7da3273" (UID: "7b74dc93-39fa-4e73-97d5-3996c7da3273"). InnerVolumeSpecName "kube-api-access-kd5nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:26.570035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.566140 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs" pod "7b74dc93-39fa-4e73-97d5-3996c7da3273" (UID: "7b74dc93-39fa-4e73-97d5-3996c7da3273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:26.650971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.650928 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.650971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.650972 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b74dc93-39fa-4e73-97d5-3996c7da3273-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.651398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.650989 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5nl\" (UniqueName: \"kubernetes.io/projected/7b74dc93-39fa-4e73-97d5-3996c7da3273-kube-api-access-kd5nl\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.651398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.651007 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b74dc93-39fa-4e73-97d5-3996c7da3273-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.771274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:26.771238 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:27.195655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.195554 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" event={"ID":"7b74dc93-39fa-4e73-97d5-3996c7da3273","Type":"ContainerDied","Data":"59e4f33c134aab2e72de9c944224486020569221ada56dc7f93aea7fe8fdaa9d"} Nov 01 22:31:27.195655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.195575 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8" Nov 01 22:31:27.195655 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.195628 5036 scope.go:122] "RemoveContainer" containerID="7320cf6acc35d17dab50635b891bce18237ea2a80f1fad1efab9b133f79c2f23" Nov 01 22:31:27.208040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.208010 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:27.266419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.266376 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:31:27.268795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.268763 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-521734-674bcd7899-xclh8"] Nov 01 22:31:27.612267 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:27.612229 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:31:28.027163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.027124 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.157391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.157329 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.363558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.347112 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.363558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.356752 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 22:31:28.363558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.357117 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" containerID="cri-o://647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada" gracePeriod=30 Nov 01 22:31:28.441352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.441315 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b74dc93-39fa-4e73-97d5-3996c7da3273" path="/var/lib/kubelet/pods/7b74dc93-39fa-4e73-97d5-3996c7da3273/volumes" Nov 01 22:31:28.759663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:28.759554 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:29.101245 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.093211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:29.368614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.368519 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" containerID="cri-o://5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1" gracePeriod=45 Nov 01 22:31:29.582557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.582078 5036 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:31:29.582557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.582167 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:31:29.646322 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.646292 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:31:29.711359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711324 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711439 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711563 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711657 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:29.711770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711728 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\" (UniqueName: \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711792 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\") pod \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\" (UID: \"0c43457f-3c85-4291-b7fc-6e7918fbfbf1\") " Nov 01 22:31:29.711903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711820 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:29.711903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.711863 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:29.712342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.712310 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:29.712342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.712343 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:29.712567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.712361 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:29.713723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.713694 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:29.713831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.713796 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:29.713831 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.713806 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8" pod "0c43457f-3c85-4291-b7fc-6e7918fbfbf1" (UID: "0c43457f-3c85-4291-b7fc-6e7918fbfbf1"). InnerVolumeSpecName "kube-api-access-nbzq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:29.813101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.813067 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbzq8\" (UniqueName: \"kubernetes.io/projected/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-kube-api-access-nbzq8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:29.813101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.813095 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:29.813101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:29.813105 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c43457f-3c85-4291-b7fc-6e7918fbfbf1-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:30.027780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.027745 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:30.227895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.227861 5036 generic.go:356] "Generic (PLEG): container finished" podID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerID="5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1" exitCode=0 Nov 01 22:31:30.228333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.227898 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" Nov 01 22:31:30.228333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.227934 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" event={"ID":"0c43457f-3c85-4291-b7fc-6e7918fbfbf1","Type":"ContainerDied","Data":"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1"} Nov 01 22:31:30.228333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.227980 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd" event={"ID":"0c43457f-3c85-4291-b7fc-6e7918fbfbf1","Type":"ContainerDied","Data":"73f1bdb5be94e35a48ee20e9a4beb1550f5032eb7733cd178aa76964647c69cb"} Nov 01 22:31:30.228333 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.228002 5036 scope.go:122] "RemoveContainer" containerID="5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1" Nov 01 22:31:30.244597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.241852 5036 scope.go:122] "RemoveContainer" containerID="5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1" Nov 01 22:31:30.244597 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:30.243154 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1\": container with ID starting with 5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1 not found: ID does not exist" containerID="5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1" Nov 01 22:31:30.244597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.243189 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1"} err="failed to get container status \"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1\": rpc error: code = NotFound desc = could not find container \"5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1\": container with ID starting with 5ca7d351cc5074e1ddbf013bcd5a1fa93c549299a23156463b83327816cb69e1 not found: ID does not exist" Nov 01 22:31:30.255666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.255646 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:31:30.259241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.259216 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d2832c/rate-test-model-d2832c-kserve-5cd655cdd9-5r7sd"] Nov 01 22:31:30.439872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.439844 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" path="/var/lib/kubelet/pods/0c43457f-3c85-4291-b7fc-6e7918fbfbf1/volumes" Nov 01 22:31:30.646937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.646912 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 22:31:30.724912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.724880 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\") pod \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " Nov 01 22:31:30.725098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.724971 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\" (UniqueName: \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\") pod \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\" (UID: \"cb5913db-ddc3-4547-bb1d-c4a908c318e7\") " Nov 01 22:31:30.725341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.725316 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume" pod "cb5913db-ddc3-4547-bb1d-c4a908c318e7" (UID: "cb5913db-ddc3-4547-bb1d-c4a908c318e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:30.725583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.725567 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5913db-ddc3-4547-bb1d-c4a908c318e7-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:30.727679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.727653 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql" pod "cb5913db-ddc3-4547-bb1d-c4a908c318e7" (UID: "cb5913db-ddc3-4547-bb1d-c4a908c318e7"). InnerVolumeSpecName "kube-api-access-hv2ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:30.798454 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.798413 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:30.828826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:30.828779 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2ql\" (UniqueName: \"kubernetes.io/projected/cb5913db-ddc3-4547-bb1d-c4a908c318e7-kube-api-access-hv2ql\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:31.239043 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.239005 5036 generic.go:356] "Generic (PLEG): container finished" podID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerID="ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f" exitCode=0 Nov 01 22:31:31.239588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.239064 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" Nov 01 22:31:31.239588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.239088 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" event={"ID":"cb5913db-ddc3-4547-bb1d-c4a908c318e7","Type":"ContainerDied","Data":"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f"} Nov 01 22:31:31.239588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.239156 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64" event={"ID":"cb5913db-ddc3-4547-bb1d-c4a908c318e7","Type":"ContainerDied","Data":"4cae792eb3d4531d997cd25a60500eedc25be9b552da6adcf7736efd11826864"} Nov 01 22:31:31.239588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.239186 5036 scope.go:122] "RemoveContainer" containerID="ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f" Nov 01 22:31:31.252159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.252135 5036 scope.go:122] "RemoveContainer" containerID="ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f" Nov 01 22:31:31.252483 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:31.252460 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f\": container with ID starting with ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f not found: ID does not exist" containerID="ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f" Nov 01 22:31:31.252584 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.252490 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f"} err="failed to get container status \"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f\": rpc error: code = NotFound desc = could not find container \"ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f\": container with ID starting with ba7343d7761f0d1aa00c42c23b16d9bc25692b9e81bf1b79fa40ee91a43dbe3f not found: ID does not exist" Nov 01 22:31:31.268882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.268858 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 22:31:31.272569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.272544 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b0b7a3-585889448c-vsh64"] Nov 01 22:31:31.548016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:31.547976 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.107076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.107042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.236105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.236073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.384980 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.384886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.515929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.515878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.517418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.517376 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" path="/var/lib/kubelet/pods/cb5913db-ddc3-4547-bb1d-c4a908c318e7/volumes" Nov 01 22:31:32.646648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:32.646567 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.063905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.063867 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.192471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.192408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.403678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.403646 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.540470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.537859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.712286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.712248 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:31:33.712544 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.712493 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" containerID="cri-o://325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8" gracePeriod=30 Nov 01 22:31:33.721353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.721313 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:31:33.721686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.721608 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" containerID="cri-o://2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5" gracePeriod=30 Nov 01 22:31:33.729383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.729358 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:31:33.730090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.729562 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" containerID="cri-o://10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0" gracePeriod=30 Nov 01 22:31:33.739633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.739610 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:31:33.740065 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.740001 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" containerID="cri-o://29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2" gracePeriod=30 Nov 01 22:31:33.863471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.847749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.927988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:33.927949 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 22:31:34.040819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.040787 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.040819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.040823 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.040859 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.040906 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.040980 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041082 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041103 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:34.041403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041123 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:34.041403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041403 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041395 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:34.041750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041387 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\") pod \"ff83270d-a1d8-4357-b365-a5eac7061e66\" (UID: \"ff83270d-a1d8-4357-b365-a5eac7061e66\") " Nov 01 22:31:34.041750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041416 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:34.041750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.041437 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:34.042137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.042110 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.042231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.042143 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.042231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.042160 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff83270d-a1d8-4357-b365-a5eac7061e66-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.042231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.042176 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.042231 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.042190 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.043477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.043428 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:34.043587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.043525 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:34.043635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.043604 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:34.043635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.043625 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt" pod "ff83270d-a1d8-4357-b365-a5eac7061e66" (UID: "ff83270d-a1d8-4357-b365-a5eac7061e66"). InnerVolumeSpecName "kube-api-access-rzjdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:34.143385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.143358 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.143385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.143383 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.143601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.143393 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff83270d-a1d8-4357-b365-a5eac7061e66-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.143601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.143405 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjdt\" (UniqueName: \"kubernetes.io/projected/ff83270d-a1d8-4357-b365-a5eac7061e66-kube-api-access-rzjdt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.173945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.173517 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:34.332562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.332523 5036 generic.go:356] "Generic (PLEG): container finished" podID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerID="647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada" exitCode=0 Nov 01 22:31:34.332753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.332583 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" Nov 01 22:31:34.332753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.332606 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" event={"ID":"ff83270d-a1d8-4357-b365-a5eac7061e66","Type":"ContainerDied","Data":"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada"} Nov 01 22:31:34.332753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.332661 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv" event={"ID":"ff83270d-a1d8-4357-b365-a5eac7061e66","Type":"ContainerDied","Data":"1fb15ae7420af11857595d238c14e7bd15670688cd9095b47fe3cb0f4152b459"} Nov 01 22:31:34.332753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.332689 5036 scope.go:122] "RemoveContainer" containerID="647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada" Nov 01 22:31:34.344832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.344813 5036 scope.go:122] "RemoveContainer" containerID="647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada" Nov 01 22:31:34.345156 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:34.345123 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada\": container with ID starting with 647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada not found: ID does not exist" containerID="647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada" Nov 01 22:31:34.345260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.345164 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada"} err="failed to get container status \"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada\": rpc error: code = NotFound desc = could not find container \"647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada\": container with ID starting with 647c067442fd75d734250e90029d1ec4c66b45fe8207a2f0bb885cc8828f3ada not found: ID does not exist" Nov 01 22:31:34.358736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.358712 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 22:31:34.365911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.365888 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-gcprv"] Nov 01 22:31:34.438543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.438480 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" path="/var/lib/kubelet/pods/ff83270d-a1d8-4357-b365-a5eac7061e66/volumes" Nov 01 22:31:34.987287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:34.987248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:35.922226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:35.922189 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:36.632320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:36.632281 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:37.435717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:37.435680 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:37.729456 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:37.729359 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:37.880998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:37.880959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:38.088996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.088957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:38.350602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.350528 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:38.390497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.390407 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" podUID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" containerName="main" containerID="cri-o://fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26" gracePeriod=45 Nov 01 22:31:38.529566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.529515 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:38.671610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.671583 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:31:38.810183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810153 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810259 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810281 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\" (UniqueName: \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810370 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810522 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\" (UniqueName: \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\") pod \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\" (UID: \"c834ace4-3421-4ae4-8b0f-3e9c55543c35\") " Nov 01 22:31:38.810684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.810684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810586 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.810869 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.810757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.811112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.811093 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.811112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.811117 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.811249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.811132 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.812882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.812854 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.813155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.813131 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "kube-api-access-hmtvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:38.813578 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.813557 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs" pod "c834ace4-3421-4ae4-8b0f-3e9c55543c35" (UID: "c834ace4-3421-4ae4-8b0f-3e9c55543c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:38.916508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.912278 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtvr\" (UniqueName: \"kubernetes.io/projected/c834ace4-3421-4ae4-8b0f-3e9c55543c35-kube-api-access-hmtvr\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.916508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.912318 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c834ace4-3421-4ae4-8b0f-3e9c55543c35-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.916508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:38.912334 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c834ace4-3421-4ae4-8b0f-3e9c55543c35-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.056597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.056562 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:31:39.090495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.086036 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:31:39.176152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.176128 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:31:39.182988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.182967 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215535 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215615 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215701 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215762 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215817 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215880 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.215916 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.216051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216010 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.216675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216472 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.216675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216496 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.216805 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216741 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.216938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216915 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.217040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216970 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.216999 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217039 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.217175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217090 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217146 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.217392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217177 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217215 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217247 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\") pod \"f40f9883-ac80-4467-a8a4-2d9009f84956\" (UID: \"f40f9883-ac80-4467-a8a4-2d9009f84956\") " Nov 01 22:31:39.217392 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217279 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\") pod \"729ec215-cbdc-4eaf-b811-3dad6db7046c\" (UID: \"729ec215-cbdc-4eaf-b811-3dad6db7046c\") " Nov 01 22:31:39.217687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.217620 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218002 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218028 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218043 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729ec215-cbdc-4eaf-b811-3dad6db7046c-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218058 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218572 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218659 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218753 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.218773 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218761 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.219413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.218823 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.219623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.219583 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.220411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.220369 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.220542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.220426 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.220542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.220512 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.220807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.220763 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.221905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.221875 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b" pod "f40f9883-ac80-4467-a8a4-2d9009f84956" (UID: "f40f9883-ac80-4467-a8a4-2d9009f84956"). InnerVolumeSpecName "kube-api-access-fqz6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.222019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.221969 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.222690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.222664 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj" pod "729ec215-cbdc-4eaf-b811-3dad6db7046c" (UID: "729ec215-cbdc-4eaf-b811-3dad6db7046c"). InnerVolumeSpecName "kube-api-access-g6mkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.294756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.294719 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.336996 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337194 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337230 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337263 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337317 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337343 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337379 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337436 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337484 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337529 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337564 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337617 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337646 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337692 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337732 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337782 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\") pod \"61197dc0-cd57-4275-80a4-8a72b298d957\" (UID: \"61197dc0-cd57-4275-80a4-8a72b298d957\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.337811 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\") pod \"1a733dee-efb9-4d83-be90-d6c17d8e1243\" (UID: \"1a733dee-efb9-4d83-be90-d6c17d8e1243\") " Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338388 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338412 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338426 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338456 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338471 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mkj\" (UniqueName: \"kubernetes.io/projected/729ec215-cbdc-4eaf-b811-3dad6db7046c-kube-api-access-g6mkj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338485 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338498 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqz6b\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-kube-api-access-fqz6b\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338511 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338525 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338540 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338554 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9883-ac80-4467-a8a4-2d9009f84956-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338570 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729ec215-cbdc-4eaf-b811-3dad6db7046c-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338583 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729ec215-cbdc-4eaf-b811-3dad6db7046c-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.343697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.338596 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9883-ac80-4467-a8a4-2d9009f84956-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.348091 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.348418 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.349035 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.349212 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.349278 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.349581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.349504 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.350021 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.349714 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.350122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.350086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.350383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.350362 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.350709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.350679 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.355573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.355542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.355699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.355651 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.357753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.357714 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.358944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.358912 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.359806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.359765 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.359907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.359785 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.359907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.359814 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm" pod "1a733dee-efb9-4d83-be90-d6c17d8e1243" (UID: "1a733dee-efb9-4d83-be90-d6c17d8e1243"). InnerVolumeSpecName "kube-api-access-vwmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.362085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.362045 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz" pod "61197dc0-cd57-4275-80a4-8a72b298d957" (UID: "61197dc0-cd57-4275-80a4-8a72b298d957"). InnerVolumeSpecName "kube-api-access-n8rrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.412894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.412851 5036 generic.go:356] "Generic (PLEG): container finished" podID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerID="325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8" exitCode=0 Nov 01 22:31:39.413498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.413092 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" Nov 01 22:31:39.413498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.413118 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" event={"ID":"f40f9883-ac80-4467-a8a4-2d9009f84956","Type":"ContainerDied","Data":"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8"} Nov 01 22:31:39.413498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.413188 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l" event={"ID":"f40f9883-ac80-4467-a8a4-2d9009f84956","Type":"ContainerDied","Data":"f4b4119305a27cb5cdb0c168038fb2bf7b02b9ff7e912b8ac54f6d6978283f6b"} Nov 01 22:31:39.413498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.413222 5036 scope.go:122] "RemoveContainer" containerID="325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8" Nov 01 22:31:39.416795 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.416633 5036 generic.go:356] "Generic (PLEG): container finished" podID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerID="29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2" exitCode=0 Nov 01 22:31:39.416948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.416824 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" event={"ID":"1a733dee-efb9-4d83-be90-d6c17d8e1243","Type":"ContainerDied","Data":"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2"} Nov 01 22:31:39.416948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.416869 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" event={"ID":"1a733dee-efb9-4d83-be90-d6c17d8e1243","Type":"ContainerDied","Data":"d786691b19b8864eafef7f1b1dbeeb304ab4dd272a7cb8e5d8e3bacec9f32247"} Nov 01 22:31:39.416948 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.416943 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9" Nov 01 22:31:39.428408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.428351 5036 generic.go:356] "Generic (PLEG): container finished" podID="61197dc0-cd57-4275-80a4-8a72b298d957" containerID="10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0" exitCode=0 Nov 01 22:31:39.428408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.428379 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" event={"ID":"61197dc0-cd57-4275-80a4-8a72b298d957","Type":"ContainerDied","Data":"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0"} Nov 01 22:31:39.428408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.428404 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" Nov 01 22:31:39.428600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.428430 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk" event={"ID":"61197dc0-cd57-4275-80a4-8a72b298d957","Type":"ContainerDied","Data":"851cc91fcd0807e16dcad35e1a4d7af4fce2f1ab5abfe11f2e35d9ac6d8d5093"} Nov 01 22:31:39.430436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.430415 5036 generic.go:356] "Generic (PLEG): container finished" podID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerID="2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5" exitCode=0 Nov 01 22:31:39.430590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.430496 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" Nov 01 22:31:39.430590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.430501 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" event={"ID":"729ec215-cbdc-4eaf-b811-3dad6db7046c","Type":"ContainerDied","Data":"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5"} Nov 01 22:31:39.430590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.430547 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj" event={"ID":"729ec215-cbdc-4eaf-b811-3dad6db7046c","Type":"ContainerDied","Data":"809b42c3cd408e9d46a4771cfafe3c800cda1c947111a9990b7057ccafbea833"} Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.432040 5036 generic.go:356] "Generic (PLEG): container finished" podID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" containerID="fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26" exitCode=0 Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.432071 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" event={"ID":"c834ace4-3421-4ae4-8b0f-3e9c55543c35","Type":"ContainerDied","Data":"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26"} Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.432096 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" event={"ID":"c834ace4-3421-4ae4-8b0f-3e9c55543c35","Type":"ContainerDied","Data":"f599aa343f6d1b9adfa47b5cda15d7a2167a189197dbe6b23a435384ba2e0129"} Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.432143 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439256 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439281 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439319 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61197dc0-cd57-4275-80a4-8a72b298d957-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439328 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439338 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439361 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439372 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439385 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439398 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmdm\" (UniqueName: \"kubernetes.io/projected/1a733dee-efb9-4d83-be90-d6c17d8e1243-kube-api-access-vwmdm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439412 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439425 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a733dee-efb9-4d83-be90-d6c17d8e1243-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439438 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439485 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439497 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61197dc0-cd57-4275-80a4-8a72b298d957-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439522 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439536 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61197dc0-cd57-4275-80a4-8a72b298d957-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439548 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a733dee-efb9-4d83-be90-d6c17d8e1243-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.440455 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.439560 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rrz\" (UniqueName: \"kubernetes.io/projected/61197dc0-cd57-4275-80a4-8a72b298d957-kube-api-access-n8rrz\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.458296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.458261 5036 scope.go:122] "RemoveContainer" containerID="325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8" Nov 01 22:31:39.458608 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:39.458585 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8\": container with ID starting with 325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8 not found: ID does not exist" containerID="325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8" Nov 01 22:31:39.458709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.458614 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8"} err="failed to get container status \"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8\": rpc error: code = NotFound desc = could not find container \"325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8\": container with ID starting with 325454a8c46391fd0142869fc0200ecd33206b04937c5838fc6f8140a60600a8 not found: ID does not exist" Nov 01 22:31:39.458709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.458630 5036 scope.go:122] "RemoveContainer" containerID="29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2" Nov 01 22:31:39.469345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.469323 5036 scope.go:122] "RemoveContainer" containerID="29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2" Nov 01 22:31:39.469691 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:39.469672 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2\": container with ID starting with 29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2 not found: ID does not exist" containerID="29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2" Nov 01 22:31:39.469746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.469697 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2"} err="failed to get container status \"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2\": rpc error: code = NotFound desc = could not find container \"29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2\": container with ID starting with 29a6fcf547e634e020d8062b5f361f9d01a333084164e41a47f63834299919c2 not found: ID does not exist" Nov 01 22:31:39.469746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.469712 5036 scope.go:122] "RemoveContainer" containerID="10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0" Nov 01 22:31:39.477113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.477092 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:31:39.480741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.480714 5036 scope.go:122] "RemoveContainer" containerID="10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0" Nov 01 22:31:39.481049 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:39.481016 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0\": container with ID starting with 10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0 not found: ID does not exist" containerID="10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0" Nov 01 22:31:39.481152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.481049 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0"} err="failed to get container status \"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0\": rpc error: code = NotFound desc = could not find container \"10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0\": container with ID starting with 10f23f6f560a388cf363cf6819e218b20f2ac7813e99b5863b88ac09d91f82a0 not found: ID does not exist" Nov 01 22:31:39.481152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.481071 5036 scope.go:122] "RemoveContainer" containerID="2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5" Nov 01 22:31:39.483420 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.483398 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-nh2cj"] Nov 01 22:31:39.493329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.493306 5036 scope.go:122] "RemoveContainer" containerID="2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5" Nov 01 22:31:39.493638 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:39.493615 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5\": container with ID starting with 2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5 not found: ID does not exist" containerID="2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5" Nov 01 22:31:39.493707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.493650 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5"} err="failed to get container status \"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5\": rpc error: code = NotFound desc = could not find container \"2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5\": container with ID starting with 2a6019411208a8a69a097641949915973736b60c7fbbfe9cf8b393c5ceb7d0d5 not found: ID does not exist" Nov 01 22:31:39.493707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.493670 5036 scope.go:122] "RemoveContainer" containerID="fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26" Nov 01 22:31:39.514149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.508049 5036 scope.go:122] "RemoveContainer" containerID="fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26" Nov 01 22:31:39.514149 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:39.508603 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26\": container with ID starting with fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26 not found: ID does not exist" containerID="fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26" Nov 01 22:31:39.514149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.508638 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26"} err="failed to get container status \"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26\": rpc error: code = NotFound desc = could not find container \"fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26\": container with ID starting with fe43e93f591197f368c1296882e54233b47387789987ce52918fcdeece534f26 not found: ID does not exist" Nov 01 22:31:39.514149 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.513699 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:31:39.519278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.519260 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-twzg9"] Nov 01 22:31:39.523227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.523203 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:31:39.526697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.526669 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-v472l"] Nov 01 22:31:39.536457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.536423 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:31:39.542895 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.542875 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-96xpk"] Nov 01 22:31:39.553924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.553902 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:31:39.558320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:39.558298 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-09656e/ipp-route-09656e-kserve-556984d957-gv6vt"] Nov 01 22:31:40.245034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.241323 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:40.441124 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.441089 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" path="/var/lib/kubelet/pods/1a733dee-efb9-4d83-be90-d6c17d8e1243/volumes" Nov 01 22:31:40.441770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.441740 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" path="/var/lib/kubelet/pods/61197dc0-cd57-4275-80a4-8a72b298d957/volumes" Nov 01 22:31:40.442311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.442293 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" path="/var/lib/kubelet/pods/729ec215-cbdc-4eaf-b811-3dad6db7046c/volumes" Nov 01 22:31:40.443212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.443192 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" path="/var/lib/kubelet/pods/c834ace4-3421-4ae4-8b0f-3e9c55543c35/volumes" Nov 01 22:31:40.443739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.443723 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" path="/var/lib/kubelet/pods/f40f9883-ac80-4467-a8a4-2d9009f84956/volumes" Nov 01 22:31:40.747909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.747874 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:40.883001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:40.882966 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:41.024090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.024048 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:31:41.024336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.024308 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" containerID="cri-o://b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218" gracePeriod=30 Nov 01 22:31:41.024678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.024650 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:31:41.024875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.024851 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" containerID="cri-o://2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794" gracePeriod=30 Nov 01 22:31:41.031008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.030601 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:31:41.031008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.030782 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" containerID="cri-o://b4ec59eb5560cc71fd894eb07f88a9b524679f2fd368f9aa8a897300876b48c3" gracePeriod=30 Nov 01 22:31:41.046917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.045751 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:41.177593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:41.177529 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:42.036147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:42.034702 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:42.762692 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:42.762653 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:43.459536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:43.455480 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:44.189475 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:44.189409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:44.397031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:44.396990 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:44.525903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:44.525862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:44.824437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:44.824337 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:44.952346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:44.952313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:45.401436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:45.395795 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:45.558466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:45.550762 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:45.795532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:45.795492 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:45.950961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:45.950927 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:46.129213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.129175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:46.337636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.337036 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:31:46.340738 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.340715 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:31:46.437942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.437907 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.437985 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438024 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438060 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438099 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438179 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438211 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438295 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438342 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438392 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438421 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438479 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438556 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438619 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\") pod \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\" (UID: \"d5178aae-9f8e-43db-9c99-56aae2a3aff5\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438676 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.438800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.438718 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\") pod \"d9b3df78-41a8-4e31-a553-7531916957f1\" (UID: \"d9b3df78-41a8-4e31-a553-7531916957f1\") " Nov 01 22:31:46.441717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.441689 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.442742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.441929 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.442742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.442358 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.442992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.442807 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.443934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.443747 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.444352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.444185 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.445205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.444400 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.445205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.444693 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "kube-api-access-6qqxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.445205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.444689 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.445967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.445213 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.445967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.445482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.445967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.445723 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.445967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.445946 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.446526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.446482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.447091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.447065 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy" pod "d9b3df78-41a8-4e31-a553-7531916957f1" (UID: "d9b3df78-41a8-4e31-a553-7531916957f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.447969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.447939 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.448787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.448757 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "kube-api-access-24dkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.449611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.449583 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo" pod "d5178aae-9f8e-43db-9c99-56aae2a3aff5" (UID: "d5178aae-9f8e-43db-9c99-56aae2a3aff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.497494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.495654 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.504819 5036 generic.go:356] "Generic (PLEG): container finished" podID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerID="2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794" exitCode=0 Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.504899 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" event={"ID":"d5178aae-9f8e-43db-9c99-56aae2a3aff5","Type":"ContainerDied","Data":"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794"} Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.504938 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" event={"ID":"d5178aae-9f8e-43db-9c99-56aae2a3aff5","Type":"ContainerDied","Data":"3e17b34950292fff0f7165f42be9d8a0b6dfbb6413a417032a61e09912e2e09e"} Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.504960 5036 scope.go:122] "RemoveContainer" containerID="2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794" Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.505073 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k" Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.509061 5036 generic.go:356] "Generic (PLEG): container finished" podID="d9b3df78-41a8-4e31-a553-7531916957f1" containerID="b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218" exitCode=0 Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.509120 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.509305 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" event={"ID":"d9b3df78-41a8-4e31-a553-7531916957f1","Type":"ContainerDied","Data":"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218"} Nov 01 22:31:46.510483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.509349 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w" event={"ID":"d9b3df78-41a8-4e31-a553-7531916957f1","Type":"ContainerDied","Data":"0e1406a7548890c8306b42efce91f1d52051a000108cbbda805c289860dc337d"} Nov 01 22:31:46.511957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.511934 5036 generic.go:356] "Generic (PLEG): container finished" podID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerID="b4ec59eb5560cc71fd894eb07f88a9b524679f2fd368f9aa8a897300876b48c3" exitCode=0 Nov 01 22:31:46.512092 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.512013 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" event={"ID":"26fb4333-23a5-4f4a-aa3b-069ef0074ba6","Type":"ContainerDied","Data":"b4ec59eb5560cc71fd894eb07f88a9b524679f2fd368f9aa8a897300876b48c3"} Nov 01 22:31:46.518778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.518756 5036 scope.go:122] "RemoveContainer" containerID="2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794" Nov 01 22:31:46.519061 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:46.519041 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794\": container with ID starting with 2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794 not found: ID does not exist" containerID="2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794" Nov 01 22:31:46.519159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.519067 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794"} err="failed to get container status \"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794\": rpc error: code = NotFound desc = could not find container \"2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794\": container with ID starting with 2a5457e9873fe819789469887739802933e60c29d320729ec9f6f0857a72d794 not found: ID does not exist" Nov 01 22:31:46.519159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.519088 5036 scope.go:122] "RemoveContainer" containerID="b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218" Nov 01 22:31:46.529422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.529400 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:31:46.532268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.532167 5036 scope.go:122] "RemoveContainer" containerID="b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218" Nov 01 22:31:46.532572 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:46.532547 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218\": container with ID starting with b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218 not found: ID does not exist" containerID="b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218" Nov 01 22:31:46.532666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.532582 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218"} err="failed to get container status \"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218\": rpc error: code = NotFound desc = could not find container \"b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218\": container with ID starting with b8d56617e37bb366e91edc2ac94fe2ac82f1a2a240a5aa041e6091c50ffa9218 not found: ID does not exist" Nov 01 22:31:46.534517 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.534459 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gnn7w"] Nov 01 22:31:46.540481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540458 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqxw\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-kube-api-access-6qqxw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540619 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b3df78-41a8-4e31-a553-7531916957f1-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540703 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540720 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540730 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540741 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540750 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540759 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b3df78-41a8-4e31-a553-7531916957f1-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540767 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540795 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540821 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540838 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5178aae-9f8e-43db-9c99-56aae2a3aff5-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540855 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540872 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b3df78-41a8-4e31-a553-7531916957f1-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540887 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b3df78-41a8-4e31-a553-7531916957f1-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540902 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540919 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dkj\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-kube-api-access-24dkj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.540933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.540936 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5178aae-9f8e-43db-9c99-56aae2a3aff5-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.550568 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.550543 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:31:46.553829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.553799 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-mq89k"] Nov 01 22:31:46.694704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.694676 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.743428 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.743509 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.743555 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.743586 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.743742 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744029 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744035 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744095 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744130 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744312 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.744624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744381 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\") pod \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\" (UID: \"26fb4333-23a5-4f4a-aa3b-069ef0074ba6\") " Nov 01 22:31:46.745435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744906 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.745435 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744928 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.746439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744312 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.746636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.744505 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.746636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.746344 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.746636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.746399 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.746636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.746547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.746967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.746943 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.749593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.749564 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt" pod "26fb4333-23a5-4f4a-aa3b-069ef0074ba6" (UID: "26fb4333-23a5-4f4a-aa3b-069ef0074ba6"). InnerVolumeSpecName "kube-api-access-sb4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846086 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846116 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb4wt\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-kube-api-access-sb4wt\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846128 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846136 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846145 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846153 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.846189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.846163 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fb4333-23a5-4f4a-aa3b-069ef0074ba6-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.849287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:46.849254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:47.525238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.525207 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" Nov 01 22:31:47.525238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.525230 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l" event={"ID":"26fb4333-23a5-4f4a-aa3b-069ef0074ba6","Type":"ContainerDied","Data":"5005266c19676b2c7915c10273d7590458f7fa4cc84cf5dece9010603d6689d2"} Nov 01 22:31:47.525510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.525290 5036 scope.go:122] "RemoveContainer" containerID="b4ec59eb5560cc71fd894eb07f88a9b524679f2fd368f9aa8a897300876b48c3" Nov 01 22:31:47.554360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.554327 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:31:47.562603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.562576 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-gb72l"] Nov 01 22:31:47.643922 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:47.643890 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:48.341402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:48.341365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:48.438873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:48.438834 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" path="/var/lib/kubelet/pods/26fb4333-23a5-4f4a-aa3b-069ef0074ba6/volumes" Nov 01 22:31:48.439335 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:48.439320 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" path="/var/lib/kubelet/pods/d5178aae-9f8e-43db-9c99-56aae2a3aff5/volumes" Nov 01 22:31:48.439778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:48.439762 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" path="/var/lib/kubelet/pods/d9b3df78-41a8-4e31-a553-7531916957f1/volumes" Nov 01 22:31:48.857229 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:31:48.857196 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:31:49.032523 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:49.032489 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:49.725012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:49.724968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:50.424151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:50.424115 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:51.257938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:51.255797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:51.771366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:51.771327 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:51.909259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:51.909217 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:52.044785 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:52.044520 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:52.194184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:52.194148 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:52.321327 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:52.321243 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:52.444123 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:52.444081 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:52.732243 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:52.732137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:53.078502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.078460 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:53.210251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.210213 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:53.317855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.317818 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:31:53.318096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.318068 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" containerID="cri-o://4dfcd7fb817b46c862f72f71aca57e893fb80f3efa925bca416570535e103432" gracePeriod=30 Nov 01 22:31:53.386189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.386105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:53.816997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.816960 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:53.949894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:53.949856 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:54.708461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:54.708411 5036 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:31:54.708934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:54.708485 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:31:54.738161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:54.738129 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:55.543216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:55.543179 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:56.235614 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:56.235573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:57.037466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:57.037423 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:57.834579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:57.834545 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:58.404673 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.404639 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:58.533985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.533920 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:58.628292 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.628206 5036 generic.go:356] "Generic (PLEG): container finished" podID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerID="4dfcd7fb817b46c862f72f71aca57e893fb80f3efa925bca416570535e103432" exitCode=0 Nov 01 22:31:58.628664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.628313 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" event={"ID":"bf6320fc-af60-4568-b57b-2efa090dc1fe","Type":"ContainerDied","Data":"4dfcd7fb817b46c862f72f71aca57e893fb80f3efa925bca416570535e103432"} Nov 01 22:31:58.690181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.690148 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:31:58.718933 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.718897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:58.748890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.748856 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 22:31:58.771201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.771167 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 22:31:58.820408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820372 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820427 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820590 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820664 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820710 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:58.820816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820725 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.820816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820789 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.821069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820812 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:58.821069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820825 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.821069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820867 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\") pod \"bf6320fc-af60-4568-b57b-2efa090dc1fe\" (UID: \"bf6320fc-af60-4568-b57b-2efa090dc1fe\") " Nov 01 22:31:58.821069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.820943 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:58.821304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821198 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:58.821398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821377 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:58.821504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821386 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.821504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821423 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.821504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821434 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.821504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.821476 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.823759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.823267 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:58.823759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.823494 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:58.823970 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.823940 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:58.824073 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.824048 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw" pod "bf6320fc-af60-4568-b57b-2efa090dc1fe" (UID: "bf6320fc-af60-4568-b57b-2efa090dc1fe"). InnerVolumeSpecName "kube-api-access-z4bkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:58.875504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.875477 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:58.922203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.922133 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.922203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.922166 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6320fc-af60-4568-b57b-2efa090dc1fe-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.922203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.922178 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4bkw\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-kube-api-access-z4bkw\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.922203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.922191 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:58.922203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:58.922200 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6320fc-af60-4568-b57b-2efa090dc1fe-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:59.639096 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.639064 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" Nov 01 22:31:59.639282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.639101 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9" event={"ID":"bf6320fc-af60-4568-b57b-2efa090dc1fe","Type":"ContainerDied","Data":"e71cdc119f4e290bd9f8911f9b3c9e382505245f7ad88d753e3f699d8c00bc7e"} Nov 01 22:31:59.639282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.639163 5036 scope.go:122] "RemoveContainer" containerID="4dfcd7fb817b46c862f72f71aca57e893fb80f3efa925bca416570535e103432" Nov 01 22:31:59.642585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.642555 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:59.667690 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.667667 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:31:59.673156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.673133 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-srtx9"] Nov 01 22:31:59.843918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.843878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:59.991905 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:31:59.991820 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:00.133284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.129288 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:32:00.133284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.129607 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" containerID="cri-o://a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b" gracePeriod=30 Nov 01 22:32:00.167018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.166987 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:00.297749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.297712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:00.438413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.438376 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" path="/var/lib/kubelet/pods/bf6320fc-af60-4568-b57b-2efa090dc1fe/volumes" Nov 01 22:32:00.557232 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.557143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:00.685988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:00.685950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:01.210294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:01.210256 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:01.331814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:01.331766 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:01.706619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:01.706580 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:02.617407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:02.617370 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:03.684635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:03.684595 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.039484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:04.039422 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 22:32:04.150045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:04.150005 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.302519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:04.302195 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.612374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:04.612294 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:05.329506 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.329468 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:05.431991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.431961 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:32:05.516896 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.516861 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517007 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517051 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517085 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517119 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517138 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517203 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517258 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517293 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517344 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\") pod \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\" (UID: \"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c\") " Nov 01 22:32:05.517722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517532 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.517722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517561 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.517722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517609 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:05.517722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517647 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.517954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517902 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.517954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517925 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.517954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517941 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.518140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517955 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.518140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.517968 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.519666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.519638 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.520055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.520027 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:05.520184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.520155 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.520266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.520177 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj" pod "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" (UID: "01c5b19f-c305-43fb-bd5b-43f6bcffbd8c"). InnerVolumeSpecName "kube-api-access-889qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.560249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.560214 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 22:32:05.570203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.570176 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 22:32:05.618703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.618671 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.618703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.618698 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.618703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.618709 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.618938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.618718 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889qj\" (UniqueName: \"kubernetes.io/projected/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c-kube-api-access-889qj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.688684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.688590 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:05.689632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.689608 5036 generic.go:356] "Generic (PLEG): container finished" podID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerID="a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b" exitCode=0 Nov 01 22:32:05.689784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.689671 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" Nov 01 22:32:05.689784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.689662 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerDied","Data":"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b"} Nov 01 22:32:05.689966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.689858 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p" event={"ID":"01c5b19f-c305-43fb-bd5b-43f6bcffbd8c","Type":"ContainerDied","Data":"ae3a70901340601e0d83986596c10626eb90127913f52dfb4e5be8bf96705689"} Nov 01 22:32:05.689966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.689895 5036 scope.go:122] "RemoveContainer" containerID="a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b" Nov 01 22:32:05.703345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.703318 5036 scope.go:122] "RemoveContainer" containerID="ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99" Nov 01 22:32:05.715155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.715118 5036 scope.go:122] "RemoveContainer" containerID="a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b" Nov 01 22:32:05.715600 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:05.715570 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b\": container with ID starting with a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b not found: ID does not exist" containerID="a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b" Nov 01 22:32:05.715723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.715612 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b"} err="failed to get container status \"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b\": rpc error: code = NotFound desc = could not find container \"a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b\": container with ID starting with a69bd1c4bd27df95a1544b40d8cfa91a5d5d663a1af8f8b257d2447c451d693b not found: ID does not exist" Nov 01 22:32:05.715723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.715635 5036 scope.go:122] "RemoveContainer" containerID="ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99" Nov 01 22:32:05.716400 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:05.716381 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99\": container with ID starting with ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99 not found: ID does not exist" containerID="ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99" Nov 01 22:32:05.716498 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.716406 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99"} err="failed to get container status \"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99\": rpc error: code = NotFound desc = could not find container \"ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99\": container with ID starting with ff608b47928138b9bd7bcabcf192ce5aed3729559880006e6d94aee1a127ae99 not found: ID does not exist" Nov 01 22:32:05.733768 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.733743 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:32:05.737645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:05.737622 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-kvt8p"] Nov 01 22:32:06.131214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:06.131170 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:06.439409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:06.439318 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" path="/var/lib/kubelet/pods/01c5b19f-c305-43fb-bd5b-43f6bcffbd8c/volumes" Nov 01 22:32:06.950236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:06.950197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:07.559840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:07.559803 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:07.693988 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:07.693949 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:07.909329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:07.909226 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:08.032784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:08.032746 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:08.284413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:08.284380 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:32:08.638168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:08.638080 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:09.040331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:09.040272 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" containerID="cri-o://b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1" gracePeriod=25 Nov 01 22:32:09.437247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:09.437156 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:10.173170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:10.173130 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:10.926618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:10.926580 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:11.056053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:11.056012 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:11.182600 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:11.182500 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:11.852511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:11.852476 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:12.644328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.644294 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h"] Nov 01 22:32:12.645008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.644983 5036 cpu_manager.go:414] "containerMap: removing container" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645011 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645034 5036 cpu_manager.go:414] "containerMap: removing container" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645042 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645057 5036 cpu_manager.go:414] "containerMap: removing container" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645065 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645079 5036 cpu_manager.go:414] "containerMap: removing container" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645086 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645097 5036 cpu_manager.go:414] "containerMap: removing container" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645105 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645116 5036 cpu_manager.go:414] "containerMap: removing container" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645124 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645136 5036 cpu_manager.go:414] "containerMap: removing container" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645143 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645161 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645169 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645188 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645196 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645207 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645214 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645226 5036 cpu_manager.go:414] "containerMap: removing container" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645234 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645249 5036 cpu_manager.go:414] "containerMap: removing container" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645256 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645269 5036 cpu_manager.go:414] "containerMap: removing container" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645277 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645287 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7b74dc93-39fa-4e73-97d5-3996c7da3273" containerName="maas-api" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645294 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b74dc93-39fa-4e73-97d5-3996c7da3273" containerName="maas-api" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645305 5036 cpu_manager.go:414] "containerMap: removing container" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645312 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645340 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a6ad7920-1e49-4002-9ca8-f717e36114bb" containerName="payload-processing" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645348 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ad7920-1e49-4002-9ca8-f717e36114bb" containerName="payload-processing" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645360 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645367 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645377 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645385 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645405 5036 cpu_manager.go:414] "containerMap: removing container" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645412 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645426 5036 cpu_manager.go:414] "containerMap: removing container" podUID="422f341f-492d-476d-b15d-5d3c9654c7d5" containerName="revoke-keys" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645433 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422f341f-492d-476d-b15d-5d3c9654c7d5" containerName="revoke-keys" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645464 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645472 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645486 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645493 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645652 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff83270d-a1d8-4357-b365-a5eac7061e66" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645668 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="422f341f-492d-476d-b15d-5d3c9654c7d5" containerName="revoke-keys" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645683 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c43457f-3c85-4291-b7fc-6e7918fbfbf1" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645693 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6320fc-af60-4568-b57b-2efa090dc1fe" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645702 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="01302463-9a3a-46cf-a162-d83e9a846249" containerName="payload-pre-processing" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645714 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fb4333-23a5-4f4a-aa3b-069ef0074ba6" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645723 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a733dee-efb9-4d83-be90-d6c17d8e1243" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645732 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645741 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c5b19f-c305-43fb-bd5b-43f6bcffbd8c" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645750 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="61197dc0-cd57-4275-80a4-8a72b298d957" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645759 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd82296e-8781-49de-8ff3-417e954b022d" containerName="main" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645769 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5178aae-9f8e-43db-9c99-56aae2a3aff5" containerName="istio-proxy" Nov 01 22:32:12.645755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645778 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5913db-ddc3-4547-bb1d-c4a908c318e7" containerName="payload-pre-processing" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645788 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb90432-17a2-41b1-9fbc-926a535dfc14" containerName="istio-proxy" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="e625cc85-0d4e-4677-a3ab-c05e6291d719" containerName="main" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645806 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="729ec215-cbdc-4eaf-b811-3dad6db7046c" containerName="istio-proxy" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645818 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b3df78-41a8-4e31-a553-7531916957f1" containerName="istio-proxy" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645826 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b74dc93-39fa-4e73-97d5-3996c7da3273" containerName="maas-api" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645834 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40f9883-ac80-4467-a8a4-2d9009f84956" containerName="istio-proxy" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645847 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ad7920-1e49-4002-9ca8-f717e36114bb" containerName="payload-processing" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645855 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aabc696-b098-4ffa-b640-7fbf8eddf008" containerName="main" Nov 01 22:32:12.647977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.645864 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c834ace4-3421-4ae4-8b0f-3e9c55543c35" containerName="main" Nov 01 22:32:12.651597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.651576 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.655279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.655246 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:12.656417 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.656396 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:12.656699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.656670 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h"] Nov 01 22:32:12.658347 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.658320 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:12.710960 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.710927 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.711152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.710986 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.711229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.711203 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s2z\" (UniqueName: \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.812362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.812322 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.812583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.812437 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s2z\" (UniqueName: \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.812583 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.812541 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.813015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.812985 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.815019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.814990 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:12.824781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:12.824753 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s2z\" (UniqueName: \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\") pod \"maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:13.002587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.002494 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:13.290371 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.290337 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h"] Nov 01 22:32:13.290767 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:32:13.290744 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0889293e_a6c4_47fb_b085_61d0a93094db.slice/crio-bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569 WatchSource:0}: Error finding container bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569: Status 404 returned error can't find the container with id bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569 Nov 01 22:32:13.608769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.608736 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:13.763000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.762957 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" event={"ID":"0889293e-a6c4-47fb-b085-61d0a93094db","Type":"ContainerStarted","Data":"87da8165222cbc6072e0c874c35490afff78ed78a90ca20ea41418d54644e97b"} Nov 01 22:32:13.763467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.763015 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" event={"ID":"0889293e-a6c4-47fb-b085-61d0a93094db","Type":"ContainerStarted","Data":"bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569"} Nov 01 22:32:13.773365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.773325 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" containerID="cri-o://a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43" gracePeriod=45 Nov 01 22:32:13.785132 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.785078 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" podStartSLOduration=1.785060599 podStartE2EDuration="1.785060599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:13.78028108 +0000 UTC m=+2887.473261566" watchObservedRunningTime="2026-11-01 22:32:13.785060599 +0000 UTC m=+2887.478041086" Nov 01 22:32:13.798037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.797923 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" containerID="cri-o://64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54" gracePeriod=45 Nov 01 22:32:13.975489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:13.971807 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Nov 01 22:32:14.097334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.097308 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:32:14.100880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.100828 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:32:14.239006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.238966 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\" (UniqueName: \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.239213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239031 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239068 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\" (UniqueName: \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.239213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239160 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.239408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\" (UniqueName: \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239299 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\" (UniqueName: \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239333 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.239408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239346 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.239635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239459 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.239635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239554 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239601 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.239753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239637 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.239753 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239688 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239777 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\") pod \"a336141c-c982-40b9-a358-6cc095f59f38\" (UID: \"a336141c-c982-40b9-a358-6cc095f59f38\") " Nov 01 22:32:14.239867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239794 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.239867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239802 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.240051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.239882 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\") pod \"23bb7067-f212-41c7-b4bb-b18d94c1953f\" (UID: \"23bb7067-f212-41c7-b4bb-b18d94c1953f\") " Nov 01 22:32:14.240116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.240487 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240457 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.240617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240496 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.240617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240516 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.240617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240532 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.240617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240551 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.240617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.240566 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.242027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.241999 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "kube-api-access-t52sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:14.242377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.242353 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.242536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.242351 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.242536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.242391 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "kube-api-access-j6xxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:14.242926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.242906 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs" pod "a336141c-c982-40b9-a358-6cc095f59f38" (UID: "a336141c-c982-40b9-a358-6cc095f59f38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:14.243016 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.242999 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs" pod "23bb7067-f212-41c7-b4bb-b18d94c1953f" (UID: "23bb7067-f212-41c7-b4bb-b18d94c1953f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:14.341998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.341966 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bb7067-f212-41c7-b4bb-b18d94c1953f-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.341998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.341998 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bb7067-f212-41c7-b4bb-b18d94c1953f-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.342174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.342009 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52sp\" (UniqueName: \"kubernetes.io/projected/23bb7067-f212-41c7-b4bb-b18d94c1953f-kube-api-access-t52sp\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.342174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.342021 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a336141c-c982-40b9-a358-6cc095f59f38-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.342174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.342030 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xxb\" (UniqueName: \"kubernetes.io/projected/a336141c-c982-40b9-a358-6cc095f59f38-kube-api-access-j6xxb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.342174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.342038 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a336141c-c982-40b9-a358-6cc095f59f38-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.348543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.348504 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:14.773405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.773374 5036 generic.go:356] "Generic (PLEG): container finished" podID="0889293e-a6c4-47fb-b085-61d0a93094db" containerID="87da8165222cbc6072e0c874c35490afff78ed78a90ca20ea41418d54644e97b" exitCode=0 Nov 01 22:32:14.773909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.773477 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" event={"ID":"0889293e-a6c4-47fb-b085-61d0a93094db","Type":"ContainerDied","Data":"87da8165222cbc6072e0c874c35490afff78ed78a90ca20ea41418d54644e97b"} Nov 01 22:32:14.775007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.774986 5036 generic.go:356] "Generic (PLEG): container finished" podID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerID="a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43" exitCode=0 Nov 01 22:32:14.775129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.775036 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" Nov 01 22:32:14.775129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.775069 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" event={"ID":"23bb7067-f212-41c7-b4bb-b18d94c1953f","Type":"ContainerDied","Data":"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43"} Nov 01 22:32:14.775129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.775109 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" event={"ID":"23bb7067-f212-41c7-b4bb-b18d94c1953f","Type":"ContainerDied","Data":"0aa70dd287eb70113cda38fdb676fad7b61c5bf36477a409ca5c390ca91b93b3"} Nov 01 22:32:14.775270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.775130 5036 scope.go:122] "RemoveContainer" containerID="a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43" Nov 01 22:32:14.776680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.776658 5036 generic.go:356] "Generic (PLEG): container finished" podID="a336141c-c982-40b9-a358-6cc095f59f38" containerID="64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54" exitCode=0 Nov 01 22:32:14.776798 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.776724 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" Nov 01 22:32:14.776868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.776816 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" event={"ID":"a336141c-c982-40b9-a358-6cc095f59f38","Type":"ContainerDied","Data":"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54"} Nov 01 22:32:14.776868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.776845 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9" event={"ID":"a336141c-c982-40b9-a358-6cc095f59f38","Type":"ContainerDied","Data":"610c8171ff465a605a58d2bedc451cc703789431fe83955286a254269a3361b3"} Nov 01 22:32:14.785894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.785873 5036 scope.go:122] "RemoveContainer" containerID="a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43" Nov 01 22:32:14.786153 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:14.786132 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43\": container with ID starting with a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43 not found: ID does not exist" containerID="a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43" Nov 01 22:32:14.786237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.786165 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43"} err="failed to get container status \"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43\": rpc error: code = NotFound desc = could not find container \"a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43\": container with ID starting with a6d9f27f2cd50af11d5038fdeedbe25d5564ab4ab3c48b60de037f129e466c43 not found: ID does not exist" Nov 01 22:32:14.786237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.786191 5036 scope.go:122] "RemoveContainer" containerID="64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54" Nov 01 22:32:14.796883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.796865 5036 scope.go:122] "RemoveContainer" containerID="64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54" Nov 01 22:32:14.797212 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:14.797190 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54\": container with ID starting with 64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54 not found: ID does not exist" containerID="64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54" Nov 01 22:32:14.797296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.797217 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54"} err="failed to get container status \"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54\": rpc error: code = NotFound desc = could not find container \"64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54\": container with ID starting with 64be3ccdf7eceda079f2f7b6ef415a65de5b019d9d0b1ef5e055cda65172ce54 not found: ID does not exist" Nov 01 22:32:14.810225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.810201 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 22:32:14.819024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.819002 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5"] Nov 01 22:32:14.829582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.829556 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 22:32:14.833931 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:14.833909 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/simulator-w3-ff3e70-kserve-84cf44d4c-zckj9"] Nov 01 22:32:15.053684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:15.053592 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-2-w3-ff3e70-kserve-65776d45cd-6bkn5" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:32:15.138659 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:15.138617 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:15.938400 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:15.938370 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:16.067824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.067754 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\" (UniqueName: \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\") pod \"0889293e-a6c4-47fb-b085-61d0a93094db\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " Nov 01 22:32:16.067824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.067791 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\") pod \"0889293e-a6c4-47fb-b085-61d0a93094db\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " Nov 01 22:32:16.068023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.067854 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\") pod \"0889293e-a6c4-47fb-b085-61d0a93094db\" (UID: \"0889293e-a6c4-47fb-b085-61d0a93094db\") " Nov 01 22:32:16.068341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.068310 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca" pod "0889293e-a6c4-47fb-b085-61d0a93094db" (UID: "0889293e-a6c4-47fb-b085-61d0a93094db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:16.069911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.069886 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z" pod "0889293e-a6c4-47fb-b085-61d0a93094db" (UID: "0889293e-a6c4-47fb-b085-61d0a93094db"). InnerVolumeSpecName "kube-api-access-49s2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:16.070033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.070002 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca" pod "0889293e-a6c4-47fb-b085-61d0a93094db" (UID: "0889293e-a6c4-47fb-b085-61d0a93094db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:16.169360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.169326 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49s2z\" (UniqueName: \"kubernetes.io/projected/0889293e-a6c4-47fb-b085-61d0a93094db-kube-api-access-49s2z\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.169360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.169356 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.169360 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.169366 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0889293e-a6c4-47fb-b085-61d0a93094db-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.256293 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.256255 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:16.439870 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.439785 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" path="/var/lib/kubelet/pods/23bb7067-f212-41c7-b4bb-b18d94c1953f/volumes" Nov 01 22:32:16.440255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.440241 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a336141c-c982-40b9-a358-6cc095f59f38" path="/var/lib/kubelet/pods/a336141c-c982-40b9-a358-6cc095f59f38/volumes" Nov 01 22:32:16.796024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.795989 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" Nov 01 22:32:16.796235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.796037 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h" event={"ID":"0889293e-a6c4-47fb-b085-61d0a93094db","Type":"ContainerDied","Data":"bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569"} Nov 01 22:32:16.796235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:16.796097 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbd517a3437027c73bee77641b3b47d9c88af09f3b9e000818aaf485d3c6569" Nov 01 22:32:17.050549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:17.046687 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:17.720049 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:17.720010 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:17.834397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:17.834360 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:18.642927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:18.642892 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:18.856974 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:18.856931 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:32:18.857362 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:18.857345 5036 scope.go:122] "RemoveContainer" containerID="4b74ba212b8eef563bd8840ee48c1a9721053a58fcf5667f7b6a983eadb03502" Nov 01 22:32:18.867334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:18.867313 5036 scope.go:122] "RemoveContainer" containerID="5e2c2e21748feea21c03cd706246537ee17a54ff73af5ea215ef4f5faa81fc2f" Nov 01 22:32:18.876770 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:18.876752 5036 scope.go:122] "RemoveContainer" containerID="9b7a352cc7505613ea32ea7fc6fc1ad3c2876a654b47e893c70c458c6a6295f5" Nov 01 22:32:19.330782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:19.330739 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:20.116720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.116677 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:20.583730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.583683 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" containerID="cri-o://4af9442e9b9bba641411431c629c827ba8c5315c988933bacc03919886ad693d" gracePeriod=45 Nov 01 22:32:20.599052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.599021 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:32:20.711085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.711047 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" containerID="cri-o://2a45fd11e8a959c4208f13142a4366d4220e4170c0cb658bf87d0922d7824c22" gracePeriod=45 Nov 01 22:32:20.835657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.835618 5036 generic.go:356] "Generic (PLEG): container finished" podID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerID="2a45fd11e8a959c4208f13142a4366d4220e4170c0cb658bf87d0922d7824c22" exitCode=0 Nov 01 22:32:20.835786 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.835672 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" event={"ID":"20a5df55-6d1d-4d2b-863d-e9c939863d6a","Type":"ContainerDied","Data":"2a45fd11e8a959c4208f13142a4366d4220e4170c0cb658bf87d0922d7824c22"} Nov 01 22:32:20.837618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.837594 5036 generic.go:356] "Generic (PLEG): container finished" podID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerID="4af9442e9b9bba641411431c629c827ba8c5315c988933bacc03919886ad693d" exitCode=0 Nov 01 22:32:20.837724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.837666 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" event={"ID":"2fd260ba-f2e1-4fa0-a46f-925269a37195","Type":"ContainerDied","Data":"4af9442e9b9bba641411431c629c827ba8c5315c988933bacc03919886ad693d"} Nov 01 22:32:20.854469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.854429 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:32:20.960565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:20.960527 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:21.015925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.015899 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:32:21.025667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.025846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025831 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.025918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025868 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.025918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025910 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\" (UniqueName: \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.026055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025925 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.026055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.025972 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.026055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026008 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\") pod \"2fd260ba-f2e1-4fa0-a46f-925269a37195\" (UID: \"2fd260ba-f2e1-4fa0-a46f-925269a37195\") " Nov 01 22:32:21.026226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026136 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.026226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026152 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.026404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026369 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.026404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026397 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.026604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.026413 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.028287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.028260 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.028397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.028378 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "kube-api-access-p78cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:21.028557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.028412 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs" pod "2fd260ba-f2e1-4fa0-a46f-925269a37195" (UID: "2fd260ba-f2e1-4fa0-a46f-925269a37195"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:21.127236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127140 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\" (UniqueName: \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127236 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127204 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127244 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127282 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127416 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\") pod \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\" (UID: \"20a5df55-6d1d-4d2b-863d-e9c939863d6a\") " Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127542 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127662 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.127877 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127770 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127958 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.127984 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78cf\" (UniqueName: \"kubernetes.io/projected/2fd260ba-f2e1-4fa0-a46f-925269a37195-kube-api-access-p78cf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.128003 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd260ba-f2e1-4fa0-a46f-925269a37195-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.128020 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd260ba-f2e1-4fa0-a46f-925269a37195-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.128034 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.128286 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.128050 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.129558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.129521 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:21.129789 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.129767 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "kube-api-access-j59ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:21.129871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.129841 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs" pod "20a5df55-6d1d-4d2b-863d-e9c939863d6a" (UID: "20a5df55-6d1d-4d2b-863d-e9c939863d6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:21.235489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.234476 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a5df55-6d1d-4d2b-863d-e9c939863d6a-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.235489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.234514 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59ks\" (UniqueName: \"kubernetes.io/projected/20a5df55-6d1d-4d2b-863d-e9c939863d6a-kube-api-access-j59ks\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.235489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.234529 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a5df55-6d1d-4d2b-863d-e9c939863d6a-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:21.606098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.606047 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:21.745472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.739501 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:21.851343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.851243 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" Nov 01 22:32:21.851343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.851267 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8" event={"ID":"2fd260ba-f2e1-4fa0-a46f-925269a37195","Type":"ContainerDied","Data":"3f9f5b1e9622ec57605fd1879f43a6c3d08fa19fc815a851f0367c23b4c8893f"} Nov 01 22:32:21.851343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.851332 5036 scope.go:122] "RemoveContainer" containerID="4af9442e9b9bba641411431c629c827ba8c5315c988933bacc03919886ad693d" Nov 01 22:32:21.860230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.860191 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" event={"ID":"20a5df55-6d1d-4d2b-863d-e9c939863d6a","Type":"ContainerDied","Data":"9b78b335e7cd386fb3653bc40695160abd7d3d9ea9103be232082028f02160bd"} Nov 01 22:32:21.860348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.860299 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw" Nov 01 22:32:21.869955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.869853 5036 scope.go:122] "RemoveContainer" containerID="2a45fd11e8a959c4208f13142a4366d4220e4170c0cb658bf87d0922d7824c22" Nov 01 22:32:21.873275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.873254 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:21.890198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.890169 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 22:32:21.894014 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.893984 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/premium-w4-0def36-kserve-8db555bdd-z2jg8"] Nov 01 22:32:21.904356 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.904330 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 22:32:21.919473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:21.912741 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/simulator-w4-0def36-kserve-57d778fb89-cpdqw"] Nov 01 22:32:22.007077 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.007021 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:22.295372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.295314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:22.439936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.439896 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" path="/var/lib/kubelet/pods/20a5df55-6d1d-4d2b-863d-e9c939863d6a/volumes" Nov 01 22:32:22.440355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.440339 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" path="/var/lib/kubelet/pods/2fd260ba-f2e1-4fa0-a46f-925269a37195/volumes" Nov 01 22:32:22.718032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.717939 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:22.757941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.757674 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:32:22.762756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.762618 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" podUID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" containerName="maas-api" containerID="cri-o://55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e" gracePeriod=30 Nov 01 22:32:22.853465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.851998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:22.980396 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:22.980278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:23.198117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.196931 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:23.207493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.204646 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:32:23.262495 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.262380 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h"] Nov 01 22:32:23.267155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.267115 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-56452a-nxv7h"] Nov 01 22:32:23.283439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.281398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\") pod \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " Nov 01 22:32:23.283439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.281493 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\") pod \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " Nov 01 22:32:23.283439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.281636 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\") pod \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " Nov 01 22:32:23.283439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.281673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\" (UniqueName: \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\") pod \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\" (UID: \"72d621b6-2c69-4c46-b774-b9c8027d4a7d\") " Nov 01 22:32:23.283439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.283164 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca" pod "72d621b6-2c69-4c46-b774-b9c8027d4a7d" (UID: "72d621b6-2c69-4c46-b774-b9c8027d4a7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:23.291829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.287145 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls" pod "72d621b6-2c69-4c46-b774-b9c8027d4a7d" (UID: "72d621b6-2c69-4c46-b774-b9c8027d4a7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:23.293796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.293736 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l" pod "72d621b6-2c69-4c46-b774-b9c8027d4a7d" (UID: "72d621b6-2c69-4c46-b774-b9c8027d4a7d"). InnerVolumeSpecName "kube-api-access-csh2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:23.294389 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.294348 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs" pod "72d621b6-2c69-4c46-b774-b9c8027d4a7d" (UID: "72d621b6-2c69-4c46-b774-b9c8027d4a7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:23.324399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.324367 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:23.389466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.383113 5036 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-metrics-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:23.389466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.383142 5036 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72d621b6-2c69-4c46-b774-b9c8027d4a7d-openshift-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:23.389466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.383157 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csh2l\" (UniqueName: \"kubernetes.io/projected/72d621b6-2c69-4c46-b774-b9c8027d4a7d-kube-api-access-csh2l\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:23.389466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.383168 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72d621b6-2c69-4c46-b774-b9c8027d4a7d-maas-api-tls\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:23.456631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.456583 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:23.756141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.756097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:23.889207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.889170 5036 generic.go:356] "Generic (PLEG): container finished" podID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" containerID="55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e" exitCode=0 Nov 01 22:32:23.889382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.889221 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" Nov 01 22:32:23.889382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.889250 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" event={"ID":"72d621b6-2c69-4c46-b774-b9c8027d4a7d","Type":"ContainerDied","Data":"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e"} Nov 01 22:32:23.889382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.889308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2" event={"ID":"72d621b6-2c69-4c46-b774-b9c8027d4a7d","Type":"ContainerDied","Data":"80f4dceedc9bd28801c71d8e0c253369f11325a8edd68586ef3ad6c078244aef"} Nov 01 22:32:23.889382 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.889332 5036 scope.go:122] "RemoveContainer" containerID="55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e" Nov 01 22:32:23.901857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.901839 5036 scope.go:122] "RemoveContainer" containerID="55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e" Nov 01 22:32:23.902152 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:23.902130 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e\": container with ID starting with 55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e not found: ID does not exist" containerID="55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e" Nov 01 22:32:23.902263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.902158 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e"} err="failed to get container status \"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e\": rpc error: code = NotFound desc = could not find container \"55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e\": container with ID starting with 55774bc3012e15e3a210251ed7e7e8f7bb641596a4dad28137e2b62ed624512e not found: ID does not exist" Nov 01 22:32:23.914996 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.914973 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:32:23.917629 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:23.917609 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-56452a-67dbb77f54-qkbd2"] Nov 01 22:32:24.043288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:24.043257 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:24.437579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:24.437497 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0889293e-a6c4-47fb-b085-61d0a93094db" path="/var/lib/kubelet/pods/0889293e-a6c4-47fb-b085-61d0a93094db/volumes" Nov 01 22:32:24.437983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:24.437908 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" path="/var/lib/kubelet/pods/72d621b6-2c69-4c46-b774-b9c8027d4a7d/volumes" Nov 01 22:32:24.756546 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:24.756429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:25.449581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:25.449539 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:26.166884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:26.166837 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:26.851066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:26.851025 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:27.634380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:27.634322 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:27.720619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:27.720584 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:28.408809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:28.408775 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.139897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.139864 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.357183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.357152 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 22:32:29.406838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.406800 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.436954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.436912 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\" (UniqueName: \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\") pod \"656123f1-6928-46c7-9f4c-c668afb99662\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " Nov 01 22:32:29.437501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.436990 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\") pod \"656123f1-6928-46c7-9f4c-c668afb99662\" (UID: \"656123f1-6928-46c7-9f4c-c668afb99662\") " Nov 01 22:32:29.437501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.437360 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume" pod "656123f1-6928-46c7-9f4c-c668afb99662" (UID: "656123f1-6928-46c7-9f4c-c668afb99662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:29.437625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.437522 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656123f1-6928-46c7-9f4c-c668afb99662-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:29.439100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.439078 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj" pod "656123f1-6928-46c7-9f4c-c668afb99662" (UID: "656123f1-6928-46c7-9f4c-c668afb99662"). InnerVolumeSpecName "kube-api-access-xqdlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:29.528620 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.528569 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.539002 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.538964 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqdlj\" (UniqueName: \"kubernetes.io/projected/656123f1-6928-46c7-9f4c-c668afb99662-kube-api-access-xqdlj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:29.711606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.711501 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 22:32:29.712192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.712144 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" containerID="cri-o://0062a2239dbc4dd3099e2c6dc0668986a66f7d104830b20c03625f3461b3c246" gracePeriod=30 Nov 01 22:32:29.737637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.737600 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.937947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.937910 5036 generic.go:356] "Generic (PLEG): container finished" podID="656123f1-6928-46c7-9f4c-c668afb99662" containerID="b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1" exitCode=0 Nov 01 22:32:29.938128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.937958 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" Nov 01 22:32:29.938128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.938002 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" event={"ID":"656123f1-6928-46c7-9f4c-c668afb99662","Type":"ContainerDied","Data":"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1"} Nov 01 22:32:29.938128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.938066 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8" event={"ID":"656123f1-6928-46c7-9f4c-c668afb99662","Type":"ContainerDied","Data":"536161e467e2f5a427e2c0533743368b3f7c95b209288ed1c1e6d309fb5e5b1c"} Nov 01 22:32:29.938128 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.938101 5036 scope.go:122] "RemoveContainer" containerID="b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1" Nov 01 22:32:29.955646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.954234 5036 scope.go:122] "RemoveContainer" containerID="b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1" Nov 01 22:32:29.955830 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:29.955793 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1\": container with ID starting with b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1 not found: ID does not exist" containerID="b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1" Nov 01 22:32:29.955937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.955842 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1"} err="failed to get container status \"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1\": rpc error: code = NotFound desc = could not find container \"b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1\": container with ID starting with b473f750d0116d5fa9857ca86a32d40e0cc9db2adef6b50cc5e597204cedd5c1 not found: ID does not exist" Nov 01 22:32:29.973135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.973044 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 22:32:29.973824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.973797 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:29.977275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:29.977249 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ff3e70-6bfb58bc86-8lhs8"] Nov 01 22:32:30.436610 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:30.436574 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656123f1-6928-46c7-9f4c-c668afb99662" path="/var/lib/kubelet/pods/656123f1-6928-46c7-9f4c-c668afb99662/volumes" Nov 01 22:32:30.641871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:30.641833 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:30.814216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:30.814180 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:30.949758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:30.949722 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:31.097163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:31.097077 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 22:32:31.097411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:31.097360 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" containerID="cri-o://49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77" gracePeriod=30 Nov 01 22:32:31.158715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:31.158682 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:31.493575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:31.493475 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:32.285734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:32.285689 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:32.412951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:32.412914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:32.535357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:32.535325 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:32:32.535597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:32.535569 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" containerID="cri-o://0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5" gracePeriod=30 Nov 01 22:32:32.601730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:32.601699 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:33.023893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:33.023862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:33.149981 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:33.149947 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:34.030644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.030606 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:34.720147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.720110 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:34.859168 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.859138 5036 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:32:34.859330 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.859191 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:32:34.986666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.986593 5036 generic.go:356] "Generic (PLEG): container finished" podID="91faaff1-4004-4afa-8f65-2705df604913" containerID="0062a2239dbc4dd3099e2c6dc0668986a66f7d104830b20c03625f3461b3c246" exitCode=0 Nov 01 22:32:34.987118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.987095 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" event={"ID":"91faaff1-4004-4afa-8f65-2705df604913","Type":"ContainerDied","Data":"0062a2239dbc4dd3099e2c6dc0668986a66f7d104830b20c03625f3461b3c246"} Nov 01 22:32:34.987303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.987281 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" event={"ID":"91faaff1-4004-4afa-8f65-2705df604913","Type":"ContainerDied","Data":"b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913"} Nov 01 22:32:34.987549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.987523 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23ec8d5cb459863ee6a7d4ebb3d9f2d3ac76a03269f4551c892cd8a563a6913" Nov 01 22:32:34.999838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:34.999811 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 22:32:35.004440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004416 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004507 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004598 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004640 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004708 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.004810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004744 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.004810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004785 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.005041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004833 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.005041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004870 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\") pod \"91faaff1-4004-4afa-8f65-2705df604913\" (UID: \"91faaff1-4004-4afa-8f65-2705df604913\") " Nov 01 22:32:35.005041 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.004877 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.005237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.005211 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:35.005237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.005225 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.005352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.005241 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.005841 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.005815 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.005966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.005907 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.007566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.007536 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.007683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.007591 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:35.007683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.007675 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "kube-api-access-ncqct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:35.007803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.007782 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token" pod "91faaff1-4004-4afa-8f65-2705df604913" (UID: "91faaff1-4004-4afa-8f65-2705df604913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:35.026972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.026946 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg"] Nov 01 22:32:35.036808 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.036781 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b9d752-blwhg"] Nov 01 22:32:35.106580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106534 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91faaff1-4004-4afa-8f65-2705df604913-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106578 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106596 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106611 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106627 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqct\" (UniqueName: \"kubernetes.io/projected/91faaff1-4004-4afa-8f65-2705df604913-kube-api-access-ncqct\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106644 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91faaff1-4004-4afa-8f65-2705df604913-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.106852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.106657 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91faaff1-4004-4afa-8f65-2705df604913-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.165436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.165394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:35.167367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.167340 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:32:35.323510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.323125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:35.456006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.455970 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:35.588287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.588200 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:35.995849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:35.995763 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f" Nov 01 22:32:36.035727 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.035427 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 22:32:36.035937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.035796 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4pl9f"] Nov 01 22:32:36.419320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.419281 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:36.439743 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.439461 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be22d27-675b-4284-89cb-c522b1c2501e" path="/var/lib/kubelet/pods/8be22d27-675b-4284-89cb-c522b1c2501e/volumes" Nov 01 22:32:36.440524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.440498 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91faaff1-4004-4afa-8f65-2705df604913" path="/var/lib/kubelet/pods/91faaff1-4004-4afa-8f65-2705df604913/volumes" Nov 01 22:32:36.768545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.768521 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 22:32:36.845496 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845429 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845525 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845565 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845643 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845679 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845744 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.845934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845821 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.845934 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845904 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.846053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845954 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.846053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845957 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.846053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.845997 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\") pod \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\" (UID: \"cd4906a4-cf6d-4462-be7a-c8802c4e282a\") " Nov 01 22:32:36.846222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846097 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.846222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846123 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:36.846384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846364 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.846483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846419 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.846563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846538 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.846563 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846565 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.846851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.846580 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.847972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.847942 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:36.848147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.848126 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "kube-api-access-556d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.848297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.848274 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.848297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.848291 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token" pod "cd4906a4-cf6d-4462-be7a-c8802c4e282a" (UID: "cd4906a4-cf6d-4462-be7a-c8802c4e282a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.947734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.947661 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.947734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.947687 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556d4\" (UniqueName: \"kubernetes.io/projected/cd4906a4-cf6d-4462-be7a-c8802c4e282a-kube-api-access-556d4\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.947734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.947697 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.947734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.947706 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4906a4-cf6d-4462-be7a-c8802c4e282a-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.947734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:36.947714 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4906a4-cf6d-4462-be7a-c8802c4e282a-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.005548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.005509 5036 generic.go:356] "Generic (PLEG): container finished" podID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerID="49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77" exitCode=0 Nov 01 22:32:37.005725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.005571 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" Nov 01 22:32:37.005725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.005572 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" event={"ID":"cd4906a4-cf6d-4462-be7a-c8802c4e282a","Type":"ContainerDied","Data":"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77"} Nov 01 22:32:37.005725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.005642 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g" event={"ID":"cd4906a4-cf6d-4462-be7a-c8802c4e282a","Type":"ContainerDied","Data":"5ffdc9211fa15b8068dac5c7be888f87620d0795356a9da740c254f06c3e3ec4"} Nov 01 22:32:37.005725 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.005673 5036 scope.go:122] "RemoveContainer" containerID="49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77" Nov 01 22:32:37.023307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.023286 5036 scope.go:122] "RemoveContainer" containerID="49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77" Nov 01 22:32:37.023626 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:37.023604 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77\": container with ID starting with 49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77 not found: ID does not exist" containerID="49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77" Nov 01 22:32:37.023717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.023636 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77"} err="failed to get container status \"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77\": rpc error: code = NotFound desc = could not find container \"49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77\": container with ID starting with 49ecd90022eb09a3106f8fc53fb87fa77b872e2c6f397d66744878234fd51f77 not found: ID does not exist" Nov 01 22:32:37.033571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.033548 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 22:32:37.037418 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.037399 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-vbr9g"] Nov 01 22:32:37.172399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.172358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:37.694759 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.694726 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:37.799115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.799089 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:32:37.858404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858298 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858397 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858492 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858568 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858610 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858646 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858677 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858709 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.858999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.858748 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\") pod \"c3fa88ab-54ba-41e9-870e-47f3f8099334\" (UID: \"c3fa88ab-54ba-41e9-870e-47f3f8099334\") " Nov 01 22:32:37.860585 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.860552 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:37.860783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.860759 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:37.860871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.860817 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:37.861040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.861013 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:37.861130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.861038 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:37.861744 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.861715 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:37.862908 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.862883 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:37.863064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.863039 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "kube-api-access-d8ktn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:37.864135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.864103 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo" pod "c3fa88ab-54ba-41e9-870e-47f3f8099334" (UID: "c3fa88ab-54ba-41e9-870e-47f3f8099334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:37.937628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.937589 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:37.960380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960334 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960372 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960384 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960394 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ktn\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-kube-api-access-d8ktn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960403 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960414 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa88ab-54ba-41e9-870e-47f3f8099334-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960422 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960432 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa88ab-54ba-41e9-870e-47f3f8099334-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.960745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:37.960455 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa88ab-54ba-41e9-870e-47f3f8099334-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:38.014942 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.014903 5036 generic.go:356] "Generic (PLEG): container finished" podID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerID="0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5" exitCode=0 Nov 01 22:32:38.015118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.014956 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" Nov 01 22:32:38.015118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.014991 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerDied","Data":"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5"} Nov 01 22:32:38.015118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.015052 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch" event={"ID":"c3fa88ab-54ba-41e9-870e-47f3f8099334","Type":"ContainerDied","Data":"4b0cfc499035afd43950fb2fe294f0f811694b84773490f72469422e8fb25080"} Nov 01 22:32:38.015118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.015084 5036 scope.go:122] "RemoveContainer" containerID="0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5" Nov 01 22:32:38.026313 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.026296 5036 scope.go:122] "RemoveContainer" containerID="613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5" Nov 01 22:32:38.036526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.036508 5036 scope.go:122] "RemoveContainer" containerID="0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5" Nov 01 22:32:38.036836 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:38.036809 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5\": container with ID starting with 0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5 not found: ID does not exist" containerID="0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5" Nov 01 22:32:38.036953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.036846 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5"} err="failed to get container status \"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5\": rpc error: code = NotFound desc = could not find container \"0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5\": container with ID starting with 0a6978e2c997abcc7f146d6b8c5850b4bfcbb1516b108baf809e75edc72992f5 not found: ID does not exist" Nov 01 22:32:38.036953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.036870 5036 scope.go:122] "RemoveContainer" containerID="613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5" Nov 01 22:32:38.037194 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:38.037173 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5\": container with ID starting with 613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5 not found: ID does not exist" containerID="613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5" Nov 01 22:32:38.037275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.037200 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5"} err="failed to get container status \"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5\": rpc error: code = NotFound desc = could not find container \"613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5\": container with ID starting with 613deee1d775555d41ab54ad535192198d616184f743cdf405a8a4448aafdfa5 not found: ID does not exist" Nov 01 22:32:38.041086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.041063 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:32:38.045064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.045044 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 22:32:38.048419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.048397 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-56452a-openshift-default-f48656c47-5z2ch"] Nov 01 22:32:38.067159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.067135 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 22:32:38.172977 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.172895 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:38.438386 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.438306 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" path="/var/lib/kubelet/pods/c3fa88ab-54ba-41e9-870e-47f3f8099334/volumes" Nov 01 22:32:38.438816 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.438801 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" path="/var/lib/kubelet/pods/cd4906a4-cf6d-4462-be7a-c8802c4e282a/volumes" Nov 01 22:32:38.729536 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:38.729433 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:39.478997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:39.478967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:40.147018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:40.146980 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:40.847156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:40.847114 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:41.550058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:41.550017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:42.354138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:42.354089 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:43.196828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.196792 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:43.265056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.265024 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 22:32:43.359314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.359249 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:43.495804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.495719 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:43.720694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.720656 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf"] Nov 01 22:32:43.721159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721139 5036 cpu_manager.go:414] "containerMap: removing container" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721161 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721184 5036 cpu_manager.go:414] "containerMap: removing container" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721191 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721206 5036 cpu_manager.go:414] "containerMap: removing container" podUID="0889293e-a6c4-47fb-b085-61d0a93094db" containerName="revoke-keys" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721215 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0889293e-a6c4-47fb-b085-61d0a93094db" containerName="revoke-keys" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721231 5036 cpu_manager.go:414] "containerMap: removing container" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721238 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721249 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721257 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" Nov 01 22:32:43.721271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721270 5036 cpu_manager.go:414] "containerMap: removing container" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721277 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721296 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721304 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721317 5036 cpu_manager.go:414] "containerMap: removing container" podUID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" containerName="maas-api" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721325 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" containerName="maas-api" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721364 5036 cpu_manager.go:414] "containerMap: removing container" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721372 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721385 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721393 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721514 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721528 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d621b6-2c69-4c46-b774-b9c8027d4a7d" containerName="maas-api" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721553 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721565 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a5df55-6d1d-4d2b-863d-e9c939863d6a" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721574 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd260ba-f2e1-4fa0-a46f-925269a37195" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721585 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="656123f1-6928-46c7-9f4c-c668afb99662" containerName="payload-processing" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721595 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bb7067-f212-41c7-b4bb-b18d94c1953f" containerName="main" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721605 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="0889293e-a6c4-47fb-b085-61d0a93094db" containerName="revoke-keys" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721613 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="91faaff1-4004-4afa-8f65-2705df604913" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721621 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4906a4-cf6d-4462-be7a-c8802c4e282a" containerName="istio-proxy" Nov 01 22:32:43.721872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.721629 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="a336141c-c982-40b9-a358-6cc095f59f38" containerName="main" Nov 01 22:32:43.727088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.727065 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.733787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.733760 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:43.734011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.733949 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:43.734163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.734070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf"] Nov 01 22:32:43.735197 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.735173 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:32:43.735351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.735222 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:32:43.820378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.820337 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tvm\" (UniqueName: \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.820641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.820409 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.820641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.820551 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.921597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.921554 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tvm\" (UniqueName: \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.921814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.921671 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.921814 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.921757 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.922383 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.922346 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.924269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.924243 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:43.933910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:43.933881 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tvm\" (UniqueName: \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\") pod \"maas-api-revoke-sub-9342dece-96vwf\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:44.047781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.047748 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:44.091552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.080860 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:44.220964 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.220925 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:44.343971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.343938 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf"] Nov 01 22:32:44.345770 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:32:44.345739 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1d6a13_2f8a_4f10_b68d_75674772ffb4.slice/crio-b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd WatchSource:0}: Error finding container b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd: Status 404 returned error can't find the container with id b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd Nov 01 22:32:44.347661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.347621 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:44.476136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:44.476101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:45.036209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.036174 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:45.051031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.050992 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx"] Nov 01 22:32:45.053406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.053383 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba37df5-6chqx"] Nov 01 22:32:45.080010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.079975 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" event={"ID":"6f1d6a13-2f8a-4f10-b68d-75674772ffb4","Type":"ContainerStarted","Data":"25f227bed14893a8fcff6025391c0769d69c9d129b4b6ebbb276842db0f4b069"} Nov 01 22:32:45.080142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.080042 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" event={"ID":"6f1d6a13-2f8a-4f10-b68d-75674772ffb4","Type":"ContainerStarted","Data":"b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd"} Nov 01 22:32:45.099141 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.099089 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" podStartSLOduration=2.099076444 podStartE2EDuration="2.099076444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:45.098559457 +0000 UTC m=+2918.791539943" watchObservedRunningTime="2026-11-01 22:32:45.099076444 +0000 UTC m=+2918.792056929" Nov 01 22:32:45.295587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:45.295561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:46.089777 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:46.089736 5036 generic.go:356] "Generic (PLEG): container finished" podID="6f1d6a13-2f8a-4f10-b68d-75674772ffb4" containerID="25f227bed14893a8fcff6025391c0769d69c9d129b4b6ebbb276842db0f4b069" exitCode=0 Nov 01 22:32:46.089995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:46.089860 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" event={"ID":"6f1d6a13-2f8a-4f10-b68d-75674772ffb4","Type":"ContainerDied","Data":"25f227bed14893a8fcff6025391c0769d69c9d129b4b6ebbb276842db0f4b069"} Nov 01 22:32:46.440735 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:46.440639 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f688b46-aa57-47f7-b25a-c4934dbc1c92" path="/var/lib/kubelet/pods/1f688b46-aa57-47f7-b25a-c4934dbc1c92/volumes" Nov 01 22:32:46.665228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:46.665190 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 22:32:46.665409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:46.665389 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" containerID="cri-o://a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f" gracePeriod=30 Nov 01 22:32:47.241909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.241884 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:47.357719 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.357642 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\") pod \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " Nov 01 22:32:47.357874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.357741 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\" (UniqueName: \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\") pod \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " Nov 01 22:32:47.357874 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.357774 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\") pod \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\" (UID: \"6f1d6a13-2f8a-4f10-b68d-75674772ffb4\") " Nov 01 22:32:47.358201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.358170 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca" pod "6f1d6a13-2f8a-4f10-b68d-75674772ffb4" (UID: "6f1d6a13-2f8a-4f10-b68d-75674772ffb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:47.359683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.359654 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca" pod "6f1d6a13-2f8a-4f10-b68d-75674772ffb4" (UID: "6f1d6a13-2f8a-4f10-b68d-75674772ffb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:47.359907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.359882 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm" pod "6f1d6a13-2f8a-4f10-b68d-75674772ffb4" (UID: "6f1d6a13-2f8a-4f10-b68d-75674772ffb4"). InnerVolumeSpecName "kube-api-access-x4tvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:47.458779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.458714 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:47.458779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.458746 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:47.458779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:47.458757 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tvm\" (UniqueName: \"kubernetes.io/projected/6f1d6a13-2f8a-4f10-b68d-75674772ffb4-kube-api-access-x4tvm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:48.105279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:48.105239 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" event={"ID":"6f1d6a13-2f8a-4f10-b68d-75674772ffb4","Type":"ContainerDied","Data":"b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd"} Nov 01 22:32:48.105279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:48.105271 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf" Nov 01 22:32:48.105545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:48.105303 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f38d30043fc70c790ca639132255162ca94d6d35b7d6e991547c5511d325fd" Nov 01 22:32:48.265703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:48.265647 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" containerID="cri-o://4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be" gracePeriod=25 Nov 01 22:32:48.858750 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:48.858703 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:32:50.655899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:50.655865 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:51.926646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:51.926620 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 22:32:52.105925 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.105889 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.105940 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.105968 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106057 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106079 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106100 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106143 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106166 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106193 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\") pod \"83870513-6b8e-4205-9823-87b037b374b8\" (UID: \"83870513-6b8e-4205-9823-87b037b374b8\") " Nov 01 22:32:52.106532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106203 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:52.106532 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106509 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:52.106763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106536 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.106763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106614 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:52.106763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106645 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:52.106763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.106740 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:52.108359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.108327 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "kube-api-access-6m8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:52.108359 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.108342 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:52.108569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.108435 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:52.108569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.108547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token" pod "83870513-6b8e-4205-9823-87b037b374b8" (UID: "83870513-6b8e-4205-9823-87b037b374b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:52.135909 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.135879 5036 generic.go:356] "Generic (PLEG): container finished" podID="83870513-6b8e-4205-9823-87b037b374b8" containerID="a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f" exitCode=0 Nov 01 22:32:52.136067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.135928 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" Nov 01 22:32:52.136067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.135929 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerDied","Data":"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f"} Nov 01 22:32:52.136067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.136036 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt" event={"ID":"83870513-6b8e-4205-9823-87b037b374b8","Type":"ContainerDied","Data":"84da9438db88f202442962372e43644700c1582eb116e779c68313fe3054b5c1"} Nov 01 22:32:52.136256 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.136070 5036 scope.go:122] "RemoveContainer" containerID="a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f" Nov 01 22:32:52.147983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.147963 5036 scope.go:122] "RemoveContainer" containerID="aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8" Nov 01 22:32:52.157680 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.157660 5036 scope.go:122] "RemoveContainer" containerID="a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f" Nov 01 22:32:52.157920 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:52.157900 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f\": container with ID starting with a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f not found: ID does not exist" containerID="a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f" Nov 01 22:32:52.157998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.157933 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f"} err="failed to get container status \"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f\": rpc error: code = NotFound desc = could not find container \"a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f\": container with ID starting with a144fe911264de529169873947069f0443cf21877285a4a944e9b2a7af0fae8f not found: ID does not exist" Nov 01 22:32:52.157998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.157956 5036 scope.go:122] "RemoveContainer" containerID="aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8" Nov 01 22:32:52.158269 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:52.158230 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8\": container with ID starting with aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8 not found: ID does not exist" containerID="aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8" Nov 01 22:32:52.158269 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.158255 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8"} err="failed to get container status \"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8\": rpc error: code = NotFound desc = could not find container \"aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8\": container with ID starting with aadd218694cf6076b4950090f86f84b35c66f655eb32321e6c0a03f817d572b8 not found: ID does not exist" Nov 01 22:32:52.161694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.161673 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 22:32:52.169890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.169858 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-p4lpt"] Nov 01 22:32:52.207029 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207003 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207031 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207045 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207058 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8zd\" (UniqueName: \"kubernetes.io/projected/83870513-6b8e-4205-9823-87b037b374b8-kube-api-access-6m8zd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207074 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207086 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83870513-6b8e-4205-9823-87b037b374b8-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207099 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83870513-6b8e-4205-9823-87b037b374b8-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.207150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.207112 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83870513-6b8e-4205-9823-87b037b374b8-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:52.436579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:52.436494 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83870513-6b8e-4205-9823-87b037b374b8" path="/var/lib/kubelet/pods/83870513-6b8e-4205-9823-87b037b374b8/volumes" Nov 01 22:32:53.067235 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.067168 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" containerID="cri-o://4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa" gracePeriod=45 Nov 01 22:32:53.087492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.087462 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" containerID="cri-o://0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2" gracePeriod=45 Nov 01 22:32:53.343192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.343167 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:32:53.367686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.367660 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:32:53.517469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517419 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.517654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517483 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.517654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517520 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.517654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517562 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\" (UniqueName: \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.517654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517599 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\" (UniqueName: \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517665 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517720 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517756 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517800 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517812 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.517907 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517887 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\" (UniqueName: \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.518253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.517951 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\") pod \"2c044010-8ee6-4df8-a258-6b71c1e97e80\" (UID: \"2c044010-8ee6-4df8-a258-6b71c1e97e80\") " Nov 01 22:32:53.518253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.518253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518041 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.518253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518068 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\") pod \"2861c4e5-27aa-479a-9fc4-e0431c665724\" (UID: \"2861c4e5-27aa-479a-9fc4-e0431c665724\") " Nov 01 22:32:53.518529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518302 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.518529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518434 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.518529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518482 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.518529 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518529 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.518763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518546 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.518763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518549 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.518763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.518754 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.520106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520067 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.520226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520154 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:53.520325 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520303 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:53.520366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520319 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "kube-api-access-tqfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:53.520366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520347 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm" pod "2c044010-8ee6-4df8-a258-6b71c1e97e80" (UID: "2c044010-8ee6-4df8-a258-6b71c1e97e80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.520695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.520680 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb" pod "2861c4e5-27aa-479a-9fc4-e0431c665724" (UID: "2861c4e5-27aa-479a-9fc4-e0431c665724"). InnerVolumeSpecName "kube-api-access-c7sgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:53.619175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619107 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619135 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7sgb\" (UniqueName: \"kubernetes.io/projected/2861c4e5-27aa-479a-9fc4-e0431c665724-kube-api-access-c7sgb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619150 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619162 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2861c4e5-27aa-479a-9fc4-e0431c665724-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619175 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c044010-8ee6-4df8-a258-6b71c1e97e80-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619188 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfr2\" (UniqueName: \"kubernetes.io/projected/2c044010-8ee6-4df8-a258-6b71c1e97e80-kube-api-access-tqfr2\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619200 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2861c4e5-27aa-479a-9fc4-e0431c665724-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.619482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:53.619212 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c044010-8ee6-4df8-a258-6b71c1e97e80-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:54.152175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.152136 5036 generic.go:356] "Generic (PLEG): container finished" podID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerID="4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa" exitCode=0 Nov 01 22:32:54.152705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.152178 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" Nov 01 22:32:54.152705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.152221 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" event={"ID":"2c044010-8ee6-4df8-a258-6b71c1e97e80","Type":"ContainerDied","Data":"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa"} Nov 01 22:32:54.152705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.152265 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r" event={"ID":"2c044010-8ee6-4df8-a258-6b71c1e97e80","Type":"ContainerDied","Data":"4509ef0b6d104e3f22d990d03a9d7624703a4293b92ba608351921642f4056e5"} Nov 01 22:32:54.152705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.152292 5036 scope.go:122] "RemoveContainer" containerID="4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa" Nov 01 22:32:54.153891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.153866 5036 generic.go:356] "Generic (PLEG): container finished" podID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerID="0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2" exitCode=0 Nov 01 22:32:54.154035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.153917 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" Nov 01 22:32:54.154035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.153936 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" event={"ID":"2861c4e5-27aa-479a-9fc4-e0431c665724","Type":"ContainerDied","Data":"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2"} Nov 01 22:32:54.154035 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.153980 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" event={"ID":"2861c4e5-27aa-479a-9fc4-e0431c665724","Type":"ContainerDied","Data":"d91302013687404755d6d156447eb0e4431cf591111db0e11ca6f014f21939f4"} Nov 01 22:32:54.164266 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.163991 5036 scope.go:122] "RemoveContainer" containerID="4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa" Nov 01 22:32:54.164548 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:54.164522 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa\": container with ID starting with 4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa not found: ID does not exist" containerID="4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa" Nov 01 22:32:54.164633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.164557 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa"} err="failed to get container status \"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa\": rpc error: code = NotFound desc = could not find container \"4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa\": container with ID starting with 4979cbae886697d60124897727c1f5094b02d3d5cd9618c7a88ce1a4f49649fa not found: ID does not exist" Nov 01 22:32:54.164633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.164578 5036 scope.go:122] "RemoveContainer" containerID="0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2" Nov 01 22:32:54.173618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.173596 5036 scope.go:122] "RemoveContainer" containerID="0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2" Nov 01 22:32:54.173906 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:32:54.173887 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2\": container with ID starting with 0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2 not found: ID does not exist" containerID="0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2" Nov 01 22:32:54.173956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.173912 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2"} err="failed to get container status \"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2\": rpc error: code = NotFound desc = could not find container \"0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2\": container with ID starting with 0f45d6387a109ca54c6060b8ece23cd751a92d599e21eb80ec38755e06ed14d2 not found: ID does not exist" Nov 01 22:32:54.177685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.177662 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 22:32:54.181625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.181605 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-2-w0-1f3a6b-kserve-7c5784785c-7cs8r"] Nov 01 22:32:54.192893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.192870 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 22:32:54.199042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.199021 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf"] Nov 01 22:32:54.324473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.324428 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/simulator-w0-1f3a6b-kserve-654b54d558-m64vf" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:32:54.437176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.437104 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" path="/var/lib/kubelet/pods/2861c4e5-27aa-479a-9fc4-e0431c665724/volumes" Nov 01 22:32:54.437527 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:54.437513 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" path="/var/lib/kubelet/pods/2c044010-8ee6-4df8-a258-6b71c1e97e80/volumes" Nov 01 22:32:56.024291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:56.024244 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw"] Nov 01 22:32:56.027247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:56.027213 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579dcf8f-wdlzw"] Nov 01 22:32:56.391341 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:56.391263 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:32:56.437910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:56.437879 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575a7df3-9338-4dc6-a9bf-a16d6594d9fa" path="/var/lib/kubelet/pods/575a7df3-9338-4dc6-a9bf-a16d6594d9fa/volumes" Nov 01 22:32:57.720588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:57.720540 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:58.830852 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:58.830815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:58.982115 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:58.982063 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:59.114932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:59.114846 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:59.522832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:32:59.522795 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:01.270502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:01.270463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:01.991592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:01.991556 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:33:05.861312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:05.861269 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:05.999402 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:05.999365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:06.139580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:06.139487 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 22:33:06.139779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:06.139752 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" containerID="cri-o://051d56bb053bd96e7332364cbcb014bb4ae68606c52336dfdcc02ceb88b8c12a" gracePeriod=30 Nov 01 22:33:06.211311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:06.211259 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:07.023271 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:07.023218 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf"] Nov 01 22:33:07.026390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:07.026365 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c629600c-s96nf"] Nov 01 22:33:07.720461 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:07.720413 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:33:08.439020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.438592 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6da6937-d872-47db-92a8-abfef5ae679b" path="/var/lib/kubelet/pods/e6da6937-d872-47db-92a8-abfef5ae679b/volumes" Nov 01 22:33:08.530162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.530100 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 22:33:08.647856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.647822 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\") pod \"cf1833aa-234b-4245-b9e6-66a4938f0e87\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " Nov 01 22:33:08.648026 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.647907 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\" (UniqueName: \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\") pod \"cf1833aa-234b-4245-b9e6-66a4938f0e87\" (UID: \"cf1833aa-234b-4245-b9e6-66a4938f0e87\") " Nov 01 22:33:08.648242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.648218 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume" pod "cf1833aa-234b-4245-b9e6-66a4938f0e87" (UID: "cf1833aa-234b-4245-b9e6-66a4938f0e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:08.650151 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.650118 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd" pod "cf1833aa-234b-4245-b9e6-66a4938f0e87" (UID: "cf1833aa-234b-4245-b9e6-66a4938f0e87"). InnerVolumeSpecName "kube-api-access-pkwtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:08.749355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.749326 5036 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf1833aa-234b-4245-b9e6-66a4938f0e87-plugins-config-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:08.749355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:08.749355 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwtd\" (UniqueName: \"kubernetes.io/projected/cf1833aa-234b-4245-b9e6-66a4938f0e87-kube-api-access-pkwtd\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:09.287240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.287202 5036 generic.go:356] "Generic (PLEG): container finished" podID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerID="4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be" exitCode=0 Nov 01 22:33:09.287473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.287251 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" Nov 01 22:33:09.287473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.287308 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" event={"ID":"cf1833aa-234b-4245-b9e6-66a4938f0e87","Type":"ContainerDied","Data":"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be"} Nov 01 22:33:09.287473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.287371 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw" event={"ID":"cf1833aa-234b-4245-b9e6-66a4938f0e87","Type":"ContainerDied","Data":"529b35450d03d097116880bf824fcdd79c9386b0ad4b04de5437aab456d770f6"} Nov 01 22:33:09.287473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.287401 5036 scope.go:122] "RemoveContainer" containerID="4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be" Nov 01 22:33:09.300260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.300242 5036 scope.go:122] "RemoveContainer" containerID="4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be" Nov 01 22:33:09.300559 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:33:09.300539 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be\": container with ID starting with 4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be not found: ID does not exist" containerID="4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be" Nov 01 22:33:09.300644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.300564 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be"} err="failed to get container status \"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be\": rpc error: code = NotFound desc = could not find container \"4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be\": container with ID starting with 4396a12c375338856db77db3f9d71ea8b16ac89cc06b3ee4cfd4e11e5c5235be not found: ID does not exist" Nov 01 22:33:09.312423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.312392 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 22:33:09.316886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.316862 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f3a6b-697d96996c-df8cw"] Nov 01 22:33:09.558775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:09.558687 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:33:10.437809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:10.437768 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" path="/var/lib/kubelet/pods/cf1833aa-234b-4245-b9e6-66a4938f0e87/volumes" Nov 01 22:33:11.312474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.312314 5036 generic.go:356] "Generic (PLEG): container finished" podID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerID="051d56bb053bd96e7332364cbcb014bb4ae68606c52336dfdcc02ceb88b8c12a" exitCode=0 Nov 01 22:33:11.312474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.312378 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" event={"ID":"97fc8b6a-a114-49d3-9e5d-1e4bc121f200","Type":"ContainerDied","Data":"051d56bb053bd96e7332364cbcb014bb4ae68606c52336dfdcc02ceb88b8c12a"} Nov 01 22:33:11.418010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.417947 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 22:33:11.473012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.472981 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473027 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473063 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473114 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473148 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473182 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473214 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473241 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473547 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473260 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\") pod \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\" (UID: \"97fc8b6a-a114-49d3-9e5d-1e4bc121f200\") " Nov 01 22:33:11.473729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473545 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.473729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473649 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.473846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473782 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.473846 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.473786 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:11.474113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.474086 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.475494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.475434 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:33:11.475636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.475610 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.475834 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.475801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "kube-api-access-bxtfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:11.475941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.475888 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token" pod "97fc8b6a-a114-49d3-9e5d-1e4bc121f200" (UID: "97fc8b6a-a114-49d3-9e5d-1e4bc121f200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:11.574843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574812 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.574843 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574842 5036 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-credential-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574853 5036 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-data\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574862 5036 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-envoy\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574871 5036 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istio-podinfo\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574879 5036 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574889 5036 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-istiod-ca-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574898 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtfn\" (UniqueName: \"kubernetes.io/projected/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-kube-api-access-bxtfn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.575040 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:11.574908 5036 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fc8b6a-a114-49d3-9e5d-1e4bc121f200-workload-socket\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:12.332259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.332221 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" event={"ID":"97fc8b6a-a114-49d3-9e5d-1e4bc121f200","Type":"ContainerDied","Data":"37d6566bbfa7b78a2616970fdffffc0c6913ce5ec4a6d251bc51bd06cfb289f1"} Nov 01 22:33:12.332704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.332293 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt" Nov 01 22:33:12.332704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.332311 5036 scope.go:122] "RemoveContainer" containerID="051d56bb053bd96e7332364cbcb014bb4ae68606c52336dfdcc02ceb88b8c12a" Nov 01 22:33:12.360817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.360791 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 22:33:12.365030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.365006 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-bjrqt"] Nov 01 22:33:12.436983 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:12.436957 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" path="/var/lib/kubelet/pods/97fc8b6a-a114-49d3-9e5d-1e4bc121f200/volumes" Nov 01 22:33:14.868355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:14.868304 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:14.994650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:14.994613 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:16.447339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:16.447303 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:18.857381 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:33:18.857341 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:33:19.015994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.015964 5036 scope.go:122] "RemoveContainer" containerID="eb7c7aa75ed2b5a376975064104e569e3dd1b430ae62c07f4951485298154bac" Nov 01 22:33:19.025636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.025612 5036 scope.go:122] "RemoveContainer" containerID="0062a2239dbc4dd3099e2c6dc0668986a66f7d104830b20c03625f3461b3c246" Nov 01 22:33:19.035188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.035169 5036 scope.go:122] "RemoveContainer" containerID="fae740b97479e540e03b9db903afc5c6249a2781ea3d36ccbc03cd852ba94f96" Nov 01 22:33:19.043378 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.043354 5036 scope.go:122] "RemoveContainer" containerID="18cef4747ecf9748f9b3f0d0b192ce31a21a6afd85aa7c1bcc0f1353109bd0d9" Nov 01 22:33:19.052003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.051980 5036 scope.go:122] "RemoveContainer" containerID="91dd0c41cf7470c768b6d7c26211bf96c0a9dad07f9dd122b5b0ca4c51a89975" Nov 01 22:33:19.060227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.060207 5036 scope.go:122] "RemoveContainer" containerID="2826cde3fc985b746b858f1f0d3fdb9052626e4ba85a121167826c12ecef427c" Nov 01 22:33:19.068282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:19.068265 5036 scope.go:122] "RemoveContainer" containerID="434acf5a4d7aabe1a04ea720a4246b8ab42097b84da2dea6a00dc20b0caaf6e4" Nov 01 22:33:21.204603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:21.204569 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:22.079685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:22.079633 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:31.182787 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:31.182747 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:33:35.022975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:35.022932 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km"] Nov 01 22:33:35.026840 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:35.026811 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr"] Nov 01 22:33:35.032086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:35.032051 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1b51ca-ht7km"] Nov 01 22:33:35.034260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:35.034231 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617d2576-pjmfr"] Nov 01 22:33:36.020580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:36.020543 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h"] Nov 01 22:33:36.023575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:36.023547 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e613288-frj9h"] Nov 01 22:33:36.436646 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:36.436555 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc7db18-770c-45c8-b86d-775adceabe89" path="/var/lib/kubelet/pods/5cc7db18-770c-45c8-b86d-775adceabe89/volumes" Nov 01 22:33:36.437079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:36.437056 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2162588-5a26-4be9-ba4d-56e90335d2d8" path="/var/lib/kubelet/pods/d2162588-5a26-4be9-ba4d-56e90335d2d8/volumes" Nov 01 22:33:36.437520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:36.437501 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef19f521-ad16-4ce0-9779-36f19b3b28e9" path="/var/lib/kubelet/pods/ef19f521-ad16-4ce0-9779-36f19b3b28e9/volumes" Nov 01 22:33:41.321478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:41.321418 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:43.524771 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:43.524731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:46.776888 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.776854 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k"] Nov 01 22:33:46.777480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777438 5036 cpu_manager.go:414] "containerMap: removing container" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:33:46.777480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777478 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777507 5036 cpu_manager.go:414] "containerMap: removing container" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777515 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777525 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777540 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777564 5036 cpu_manager.go:414] "containerMap: removing container" podUID="6f1d6a13-2f8a-4f10-b68d-75674772ffb4" containerName="revoke-keys" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777571 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1d6a13-2f8a-4f10-b68d-75674772ffb4" containerName="revoke-keys" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777598 5036 cpu_manager.go:414] "containerMap: removing container" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777606 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777626 5036 cpu_manager.go:414] "containerMap: removing container" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777633 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fa88ab-54ba-41e9-870e-47f3f8099334" containerName="istio-proxy" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777646 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" Nov 01 22:33:46.777710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777653 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777758 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1d6a13-2f8a-4f10-b68d-75674772ffb4" containerName="revoke-keys" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777774 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777783 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1833aa-234b-4245-b9e6-66a4938f0e87" containerName="payload-pre-processing" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777797 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c044010-8ee6-4df8-a258-6b71c1e97e80" containerName="main" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777808 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777818 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fc8b6a-a114-49d3-9e5d-1e4bc121f200" containerName="istio-proxy" Nov 01 22:33:46.778275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.777826 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2861c4e5-27aa-479a-9fc4-e0431c665724" containerName="main" Nov 01 22:33:46.782019 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.782000 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.795665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.795638 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k"] Nov 01 22:33:46.889482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889435 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889492 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889575 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889616 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f246de-749e-4ada-81a3-765a18df9fba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889643 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889777 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prh5\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-kube-api-access-4prh5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.889898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889821 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f246de-749e-4ada-81a3-765a18df9fba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.890023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889919 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.890023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.889981 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.990878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990850 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990898 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990918 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990948 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990973 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f246de-749e-4ada-81a3-765a18df9fba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.990991 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991033 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prh5\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-kube-api-access-4prh5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991191 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f246de-749e-4ada-81a3-765a18df9fba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991331 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991357 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991508 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991620 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.991871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.991844 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f246de-749e-4ada-81a3-765a18df9fba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.993552 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.993520 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f246de-749e-4ada-81a3-765a18df9fba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:46.993693 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:46.993673 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f246de-749e-4ada-81a3-765a18df9fba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:47.003484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.003434 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:47.003705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.003685 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prh5\" (UniqueName: \"kubernetes.io/projected/90f246de-749e-4ada-81a3-765a18df9fba-kube-api-access-4prh5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tmn2k\" (UID: \"90f246de-749e-4ada-81a3-765a18df9fba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:47.098108 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.098036 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:47.381095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.381063 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k"] Nov 01 22:33:47.382378 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:33:47.382342 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f246de_749e_4ada_81a3_765a18df9fba.slice/crio-7c6abe5ea7126b1c42551901126a24e3bd68456a7df42b919e23a4ad6db8bf45 WatchSource:0}: Error finding container 7c6abe5ea7126b1c42551901126a24e3bd68456a7df42b919e23a4ad6db8bf45: Status 404 returned error can't find the container with id 7c6abe5ea7126b1c42551901126a24e3bd68456a7df42b919e23a4ad6db8bf45 Nov 01 22:33:47.384550 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.384515 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:33:47.384643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.384579 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:33:47.384643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.384606 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:33:47.559261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.559215 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" event={"ID":"90f246de-749e-4ada-81a3-765a18df9fba","Type":"ContainerStarted","Data":"d52063e9bb0699c27e8ade9470ae7ebf554eb8d6a10e8495d7330ec990050d31"} Nov 01 22:33:47.559478 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.559273 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" event={"ID":"90f246de-749e-4ada-81a3-765a18df9fba","Type":"ContainerStarted","Data":"7c6abe5ea7126b1c42551901126a24e3bd68456a7df42b919e23a4ad6db8bf45"} Nov 01 22:33:47.584306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:47.584249 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podStartSLOduration=1.5842298559999999 podStartE2EDuration="1.584229856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:47.582302903 +0000 UTC m=+2981.275283390" watchObservedRunningTime="2026-11-01 22:33:47.584229856 +0000 UTC m=+2981.277210343" Nov 01 22:33:48.101669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:48.101623 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:48.103433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:48.103399 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:48.103654 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:48.103467 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:48.856592 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:33:48.856552 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:33:49.101857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:49.101816 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:49.102315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:49.101892 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:50.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:50.098840 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:50.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:50.098902 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:51.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:51.098800 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:51.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:51.098860 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:52.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:52.098777 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:52.099467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:52.098843 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:53.099469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:53.098806 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:53.099469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:53.098866 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:54.098354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:54.098319 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.098560 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:54.098373 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 01 22:33:55.487101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:55.487068 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:55.487652 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:55.487476 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:33:55.488179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:33:55.488154 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" Nov 01 22:34:01.312921 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:01.312882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:08.034075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:08.034045 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:34:08.726708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:08.726150 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:34:18.172938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:18.172852 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:34:18.862331 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:34:18.862287 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:34:19.170971 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.170872 5036 scope.go:122] "RemoveContainer" containerID="32a149741da246ce1870e28a3c6eb99032f59c62e0792ac298766f59493e1daa" Nov 01 22:34:19.183474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.183420 5036 scope.go:122] "RemoveContainer" containerID="6072dea5a025783a1724c8ac09e2599c0893204ae9f86cf6d353700c1f2df316" Nov 01 22:34:19.193707 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.193683 5036 scope.go:122] "RemoveContainer" containerID="61dd6ad8d2bda6534f814d0bdbd81ea11f485091fd927c5677c86cd0581681fd" Nov 01 22:34:19.204813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.204792 5036 scope.go:122] "RemoveContainer" containerID="98dce5c98798a35383f50e3a1801bdec043178fc306c0016981c6effaab81bce" Nov 01 22:34:19.214170 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.214145 5036 scope.go:122] "RemoveContainer" containerID="b2ab58c78fab0ed49a0a1b139dd46984aedada5d89f3be65448284dcf01a6768" Nov 01 22:34:19.223802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.223776 5036 scope.go:122] "RemoveContainer" containerID="e28c8b07566dcc9e370053c170fdda0fdff9397a58e86e6b49dff0d65bc80724" Nov 01 22:34:19.234146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.234126 5036 scope.go:122] "RemoveContainer" containerID="1fdc8209c919232580149c6ffe35c4277df3505ef2b2376c6c17e84a008a2a64" Nov 01 22:34:19.244201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.244177 5036 scope.go:122] "RemoveContainer" containerID="4f284c4f2fc86fdc66c9e1a03a68c4fe270c1d77470265a3100695b1812021a3" Nov 01 22:34:19.253810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.253784 5036 scope.go:122] "RemoveContainer" containerID="d1260bb4cabecfbb742f9084a829f943dd4fcf0d5f1952c17002b4e6276f2e78" Nov 01 22:34:19.263217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:19.263192 5036 scope.go:122] "RemoveContainer" containerID="6ac63908d91a97813c19d5197365a1579c9a19264d321bc5960869e7275e76be" Nov 01 22:34:35.247728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:35.247690 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:34:39.125384 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:39.096674 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:34:46.847857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:46.847814 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g"] Nov 01 22:34:46.848275 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:46.848258 5036 cpu_manager.go:414] "containerMap: removing container" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:34:46.848328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:46.848279 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83870513-6b8e-4205-9823-87b037b374b8" containerName="istio-proxy" Nov 01 22:34:46.854697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:46.854665 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:46.876291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:46.876253 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g"] Nov 01 22:34:47.043904 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.043873 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.043924 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044081 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.043991 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044107 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044142 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044163 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ghs\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-kube-api-access-m8ghs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044186 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044282 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044261 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.044354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.044319 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145517 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145556 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145593 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145617 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145637 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ghs\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-kube-api-access-m8ghs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145657 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145732 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145783 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.145919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145828 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.146315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.145977 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.146315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.146007 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.146315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.146202 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.146315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.146235 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.146503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.146405 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.148027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.148002 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.148200 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.148184 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.158228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.158204 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.158687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.158663 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ghs\" (UniqueName: \"kubernetes.io/projected/ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb-kube-api-access-m8ghs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-shr8g\" (UID: \"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.182329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.182308 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:47.467575 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.467541 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g"] Nov 01 22:34:47.469172 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:34:47.469144 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6c5c17_e832_4a8a_8fe4_31082ba9f7eb.slice/crio-8f9bde4eda76b6d032790dbc0774a8eff745405e07493f86bba1ac38bcd152ec WatchSource:0}: Error finding container 8f9bde4eda76b6d032790dbc0774a8eff745405e07493f86bba1ac38bcd152ec: Status 404 returned error can't find the container with id 8f9bde4eda76b6d032790dbc0774a8eff745405e07493f86bba1ac38bcd152ec Nov 01 22:34:47.471391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.471363 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:34:47.471484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.471424 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:34:47.471484 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:47.471469 5036 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 22:34:48.006372 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.006328 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" event={"ID":"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb","Type":"ContainerStarted","Data":"d277a71bd1ef279f6c42ee74b9890004fa4c71ce411d085b75f93330e66c3694"} Nov 01 22:34:48.006809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.006388 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" event={"ID":"ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb","Type":"ContainerStarted","Data":"8f9bde4eda76b6d032790dbc0774a8eff745405e07493f86bba1ac38bcd152ec"} Nov 01 22:34:48.032078 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.032023 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podStartSLOduration=2.032010909 podStartE2EDuration="2.032010909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:48.030620775 +0000 UTC m=+3041.723601262" watchObservedRunningTime="2026-11-01 22:34:48.032010909 +0000 UTC m=+3041.724991395" Nov 01 22:34:48.189616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.189571 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.189961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.189927 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:48.190377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:48.190355 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:48.857438 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:34:48.857398 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:34:49.196875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:49.196785 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.197557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:49.197513 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:50.195468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:50.188550 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.195468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:50.188609 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:51.198601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:51.198564 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.199466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:51.199201 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:52.197468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:52.194832 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.197468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:52.194894 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:53.202467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:53.186813 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.203801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:53.203236 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:54.057669 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:54.057628 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:34:54.183706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:54.183666 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-shr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.183889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:54.183726 5036 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" podUID="ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 01 22:34:55.708950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:55.708903 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:55.709619 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:55.709600 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:34:55.710836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:34:55.710812 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-shr8g" Nov 01 22:35:01.380736 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:01.380213 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:06.678161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:06.678119 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:35:18.856683 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:35:18.856648 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:35:19.345781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:19.345745 5036 scope.go:122] "RemoveContainer" containerID="cb42a2958bc97ca95eea83df6bf94dadeb7b4806d7faf6161fba611ebb8f74ea" Nov 01 22:35:19.358826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:19.358804 5036 scope.go:122] "RemoveContainer" containerID="ddbd481f70f7d5240a4bf94fb8a8a7ce2407c1eb862a475453de3fba9ae63ea9" Nov 01 22:35:19.367312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:19.367293 5036 scope.go:122] "RemoveContainer" containerID="24ef50d7fedeb4e13cb100814bc9782b0afe8b292cfeb4652d8b97d0b16dfba7" Nov 01 22:35:19.375515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:19.375498 5036 scope.go:122] "RemoveContainer" containerID="d34bbe3e1f69dc865730957708a968c4a7abda2261d810d473bffedc8d4b8b2d" Nov 01 22:35:24.222618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:24.222565 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:24.350121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:24.350076 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:30.132102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:30.132060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:30.970050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:30.970015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:35:40.022521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:40.022483 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf"] Nov 01 22:35:40.024954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:40.024927 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9298a90a-nwfnf"] Nov 01 22:35:40.436217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:40.436140 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3715fe7-9369-4523-bac3-fbe87c143814" path="/var/lib/kubelet/pods/d3715fe7-9369-4523-bac3-fbe87c143814/volumes" Nov 01 22:35:41.147210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:41.147166 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:35:45.752226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:45.752185 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:35:48.858528 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:35:48.858428 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:35:52.022827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:52.022792 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9"] Nov 01 22:35:52.025715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:52.025685 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d3c04f-xqvj9"] Nov 01 22:35:52.437003 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:35:52.436922 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44cc163-afce-48d0-9f83-5c531b9fd46f" path="/var/lib/kubelet/pods/b44cc163-afce-48d0-9f83-5c531b9fd46f/volumes" Nov 01 22:36:01.434699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:01.434627 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:01.679159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:01.679120 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:05.565775 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:05.565734 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:10.023023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:10.022986 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d"] Nov 01 22:36:10.025033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:10.024966 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f886d75-mww6d"] Nov 01 22:36:10.436717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:10.436624 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422f341f-492d-476d-b15d-5d3c9654c7d5" path="/var/lib/kubelet/pods/422f341f-492d-476d-b15d-5d3c9654c7d5/volumes" Nov 01 22:36:18.857602 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:36:18.857558 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:36:19.408788 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:19.408753 5036 scope.go:122] "RemoveContainer" containerID="05894bf1b9ebaebb321f78523e2465f0f032430f700a15d30f9cbf8a4ed657d9" Nov 01 22:36:19.419285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:19.419257 5036 scope.go:122] "RemoveContainer" containerID="6d4f381603a8371d3a1e6a33f20954fd863c6b6a36c5055ebda72d3c37ee9d18" Nov 01 22:36:19.428626 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:19.428606 5036 scope.go:122] "RemoveContainer" containerID="4f424a3309ef4ba4e66683fd37d1601bf53f6ddbeaf9c94f9f0ecc94ef3c7ee5" Nov 01 22:36:19.436302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:19.436278 5036 scope.go:122] "RemoveContainer" containerID="75342b22d7539f14aaa6602a15ab8648d2b08b1f2215d913013221905401c162" Nov 01 22:36:19.656963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:19.656924 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:24.983105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:24.983050 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:38.443355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:38.443321 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:42.200462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:42.199590 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:36:48.857062 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:36:48.857021 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:36:53.254521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.254488 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 22:36:53.255090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.254679 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" podUID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" containerName="discovery" containerID="cri-o://0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735" gracePeriod=30 Nov 01 22:36:53.518208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.518144 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 22:36:53.558569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558536 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558579 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558601 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558653 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558675 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558794 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.558994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.558946 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\") pod \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\" (UID: \"2d01a05b-4fec-4b0d-963a-2d8987b5351a\") " Nov 01 22:36:53.559146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.559120 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:53.559329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.559300 5036 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-ca-configmap\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.561153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561100 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:53.561433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561399 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:53.561581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561463 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:53.561581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561487 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:53.561702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561666 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "kube-api-access-7jzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:53.561745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.561713 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs" pod "2d01a05b-4fec-4b0d-963a-2d8987b5351a" (UID: "2d01a05b-4fec-4b0d-963a-2d8987b5351a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:36:53.660493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660458 5036 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-cacerts\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.660493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660489 5036 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-csr-dns-cert\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.660493 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660499 5036 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d01a05b-4fec-4b0d-963a-2d8987b5351a-local-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.660742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660508 5036 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-token\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.660742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660517 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzhn\" (UniqueName: \"kubernetes.io/projected/2d01a05b-4fec-4b0d-963a-2d8987b5351a-kube-api-access-7jzhn\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.660742 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.660526 5036 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d01a05b-4fec-4b0d-963a-2d8987b5351a-istio-kubeconfig\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:53.932602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.932563 5036 generic.go:356] "Generic (PLEG): container finished" podID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" containerID="0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735" exitCode=0 Nov 01 22:36:53.932791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.932616 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" Nov 01 22:36:53.932791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.932645 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" event={"ID":"2d01a05b-4fec-4b0d-963a-2d8987b5351a","Type":"ContainerDied","Data":"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735"} Nov 01 22:36:53.932791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.932725 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l" event={"ID":"2d01a05b-4fec-4b0d-963a-2d8987b5351a","Type":"ContainerDied","Data":"0be2e1c075598e5a9a5d90890fe6d73f67d31b941c313d4c1303577190c21ddf"} Nov 01 22:36:53.932791 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.932746 5036 scope.go:122] "RemoveContainer" containerID="0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735" Nov 01 22:36:53.945910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.945889 5036 scope.go:122] "RemoveContainer" containerID="0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735" Nov 01 22:36:53.946212 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:36:53.946192 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735\": container with ID starting with 0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735 not found: ID does not exist" containerID="0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735" Nov 01 22:36:53.946268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.946222 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735"} err="failed to get container status \"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735\": rpc error: code = NotFound desc = could not find container \"0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735\": container with ID starting with 0b7eec1ab54d4bc8e7560e4f6708ba66920a2c0d48ece62225526afa1d82a735 not found: ID does not exist" Nov 01 22:36:53.957018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.956994 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 22:36:53.961873 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:53.961853 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8dg8l"] Nov 01 22:36:54.437143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:54.437104 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" path="/var/lib/kubelet/pods/2d01a05b-4fec-4b0d-963a-2d8987b5351a/volumes" Nov 01 22:36:56.174641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:36:56.174605 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:01.484688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:01.484632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:06.766519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:06.766486 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:10.741656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:10.741616 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:18.856626 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:37:18.856536 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:37:19.474172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.474139 5036 scope.go:122] "RemoveContainer" containerID="5090c6f2fdfea8850a4098f0e48f4c3a287bb670f4634d4c41be60e0eb4e45fe" Nov 01 22:37:19.483262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.483240 5036 scope.go:122] "RemoveContainer" containerID="469f1cd66485670fa8a6ab5aaacbd79a13354a61679bfb1249bea4601635a18d" Nov 01 22:37:19.492055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.492019 5036 scope.go:122] "RemoveContainer" containerID="996ecc14d4b1a4a2fcda877aa454cd0d964be4b4293cb16640fd27412ed58096" Nov 01 22:37:19.500111 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.500092 5036 scope.go:122] "RemoveContainer" containerID="8eadd75bd19dd7399a899a475e3d12c06a0454986882df0c4a8de0b26f9b8b52" Nov 01 22:37:19.508540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.508519 5036 scope.go:122] "RemoveContainer" containerID="73357e963dee7ca777265db8b2ecc97b9639325323c3316edef0bbfa9fae2986" Nov 01 22:37:19.516605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.516579 5036 scope.go:122] "RemoveContainer" containerID="dc357fc0fd6ad371146be8584ccd17380c31b85bd875f6034db3eef087c21ecf" Nov 01 22:37:19.524703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:19.524680 5036 scope.go:122] "RemoveContainer" containerID="15054c286aeddd4bdaf5d05a77bf46d21c8bf39b912bdaa94e75c7802546cdf4" Nov 01 22:37:25.273711 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:25.273671 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:29.052007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:29.051967 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:34.866509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:34.866471 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:48.022250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:48.022211 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf"] Nov 01 22:37:48.027222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:48.027189 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9342dece-96vwf"] Nov 01 22:37:48.435953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:48.435870 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1d6a13-2f8a-4f10-b68d-75674772ffb4" path="/var/lib/kubelet/pods/6f1d6a13-2f8a-4f10-b68d-75674772ffb4/volumes" Nov 01 22:37:48.856601 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:37:48.856561 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:37:53.188521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:53.188480 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:37:55.995613 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:37:55.995573 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:01.534094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:38:01.534054 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:03.268411 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:38:03.268369 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:38:18.857488 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:38:18.857430 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:38:19.574651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:38:19.574619 5036 scope.go:122] "RemoveContainer" containerID="87da8165222cbc6072e0c874c35490afff78ed78a90ca20ea41418d54644e97b" Nov 01 22:38:48.858280 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:38:48.858200 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:39:01.584729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:01.584689 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:02.680288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:02.680250 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:39:06.756011 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:06.755976 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:39:08.070294 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:08.070264 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:39:08.145342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:08.145275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:08.762910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:08.762881 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:39:09.999806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:09.999764 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:10.404872 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:10.404823 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:10.579413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:10.579379 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:14.215994 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:14.215951 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:14.343012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:14.342979 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:15.435147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:15.435109 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:15.571519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:15.571484 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:15.704361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:15.704275 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:16.260702 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:16.260632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:16.956095 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:16.956064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:17.455205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:17.455170 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:39:17.770348 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:17.770306 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:18.639623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:18.639584 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:18.858285 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:39:18.858251 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:39:19.359827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:19.359791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:19.597319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:19.597283 5036 scope.go:122] "RemoveContainer" containerID="25f227bed14893a8fcff6025391c0769d69c9d129b4b6ebbb276842db0f4b069" Nov 01 22:39:20.144508 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:20.144464 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:20.856724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:20.856691 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:21.665699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:21.665639 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:22.365954 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:22.365898 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:23.161025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:23.160967 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:23.963138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:23.963101 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:24.838114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:24.838073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:25.686555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:25.686482 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:26.542161 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:26.542122 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:27.253969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:27.253933 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:27.944704 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:27.944662 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:28.853291 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:28.853253 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:29.153085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:29.152992 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:29.811625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:29.811587 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:30.007018 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.006982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:30.183689 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.183594 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:30.607663 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.607630 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:39:30.608070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.608056 5036 cpu_manager.go:414] "containerMap: removing container" podUID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" containerName="discovery" Nov 01 22:39:30.608120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.608072 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" containerName="discovery" Nov 01 22:39:30.608162 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.608145 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d01a05b-4fec-4b0d-963a-2d8987b5351a" containerName="discovery" Nov 01 22:39:30.612914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.612890 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.627511 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.627485 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-096081\"/\"simulator-single-main-096081-kserve-self-signed-certs\"" Nov 01 22:39:30.631105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.631076 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-096081\"/\"default-dockercfg-cb7hh\"" Nov 01 22:39:30.633927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.633903 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:39:30.682838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.682809 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-096081\"/\"openshift-service-ca.crt\"" Nov 01 22:39:30.682838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.682832 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-096081\"/\"kube-root-ca.crt\"" Nov 01 22:39:30.691283 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.691416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691367 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.691416 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691398 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.691551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691456 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cs8\" (UniqueName: \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.691551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691528 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.691661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.691563 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.721120 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.721078 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:30.792148 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792119 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792205 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792242 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792284 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cs8\" (UniqueName: \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792313 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792333 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792690 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792766 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792714 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.792811 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.792770 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.794747 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.794722 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.795184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.795157 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.804687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.804661 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cs8\" (UniqueName: \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\") pod \"simulator-single-main-096081-kserve-6b6c87c44-p9fwm\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.935880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.935792 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:30.947940 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:30.947906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:31.223567 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.223535 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:39:31.225256 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:39:31.225210 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbda77944_7fb6_42fe_b0f7_a0d0be4f97ea.slice/crio-177df8f53268e70172c6c1ce7f984e615de11dad5d215d3f9a69efc3acbed426 WatchSource:0}: Error finding container 177df8f53268e70172c6c1ce7f984e615de11dad5d215d3f9a69efc3acbed426: Status 404 returned error can't find the container with id 177df8f53268e70172c6c1ce7f984e615de11dad5d215d3f9a69efc3acbed426 Nov 01 22:39:31.769015 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.768972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:31.839951 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.839918 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" event={"ID":"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea","Type":"ContainerStarted","Data":"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd"} Nov 01 22:39:31.840165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.839960 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" event={"ID":"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea","Type":"ContainerStarted","Data":"177df8f53268e70172c6c1ce7f984e615de11dad5d215d3f9a69efc3acbed426"} Nov 01 22:39:31.862214 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.862167 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" podStartSLOduration=1.862154516 podStartE2EDuration="1.862154516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:31.860750766 +0000 UTC m=+3325.553731257" watchObservedRunningTime="2026-11-01 22:39:31.862154516 +0000 UTC m=+3325.555134999" Nov 01 22:39:31.936783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:31.936752 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:32.563794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:32.563761 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:39:32.710385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:32.710349 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:33.669399 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:33.669368 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:34.519592 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:34.519551 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:35.431699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:35.431659 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:36.140806 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:36.140763 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:36.850397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:36.850346 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:37.547915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:37.547880 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:38.255138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:38.255087 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:38.964565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:38.964528 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:39.640959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:39.640924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:40.355032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:40.354993 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:40.936781 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:40.936737 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:40.949492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:40.949462 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:41.048618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:41.048586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:41.760929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:41.760891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:41.908259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:41.908229 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:39:42.570774 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:42.570730 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:43.262226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:43.262168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:44.132991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:44.132959 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:44.843720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:44.843682 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:45.666703 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:45.666670 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:46.353334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:46.353304 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:47.304364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:47.304326 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:48.049790 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:48.049757 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:48.772331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:48.772292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:48.857483 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:39:48.857423 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:39:49.560203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:49.560170 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:49.671179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:49.671144 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:39:49.729250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:49.729214 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:50.358140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:50.358099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:51.067055 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:51.067011 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:51.208588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:51.208546 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:51.849300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:51.849262 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:52.655935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:52.655897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:53.447730 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:53.447693 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:54.143199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:54.143157 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:54.955489 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:54.955434 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:55.652254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:55.652211 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:56.361634 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:56.361598 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:57.061992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:57.061955 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:57.758009 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:57.757972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:58.552678 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:58.552645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:59.244209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:59.244171 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:59.946597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:39:59.946559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:00.637440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:00.637399 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:01.349657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:01.349617 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:01.647807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:01.647721 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:02.168129 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:02.168086 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:02.840295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:02.840250 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:03.640956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:03.640914 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:05.587453 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:05.587417 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:05.746519 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:05.746466 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:06.483146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:06.483107 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:07.198731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:07.198673 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:07.980812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:07.980780 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:08.667830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:08.667787 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:09.564912 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:09.564873 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:10.361892 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:10.361852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:11.293668 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:11.293636 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:12.175935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:12.175897 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:12.992597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:12.992558 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:13.812091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:13.812052 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:14.082042 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:14.081950 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:14.798591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:14.798544 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:40:14.975966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:14.975924 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:15.822482 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:15.822428 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:16.740166 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:16.740129 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:17.458796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:17.458689 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:18.176187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:18.176139 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:18.856258 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:40:18.856218 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:40:18.940089 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:18.940046 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:19.044963 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:19.044925 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:40:19.147724 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:19.147638 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:19.667793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:19.667742 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:20.362228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:20.362193 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:21.035728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:21.035696 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:21.730624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:21.730585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:22.442643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:22.442601 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:23.145007 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:23.144968 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:23.847579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:23.847535 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:24.463581 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:24.463545 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:24.595233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:24.595197 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:25.119494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:25.119429 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:25.271407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:25.271365 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:25.407320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:25.407228 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:25.830658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:25.830628 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.682939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:26.682899 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:40:31.854637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:31.854590 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.984588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:31.984546 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:32.212910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:32.212809 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:34.065642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.065581 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" podUID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" containerName="main" containerID="cri-o://cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd" gracePeriod=45 Nov 01 22:40:34.314365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.314342 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:40:34.393697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393615 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393697 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393664 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393710 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393786 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\" (UniqueName: \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393855 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\") pod \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\" (UID: \"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea\") " Nov 01 22:40:34.393947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393897 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:34.394211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.393987 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:34.394211 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.394085 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:34.394336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.394212 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:34.394336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.394230 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:34.394336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.394240 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:34.396037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.396003 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:34.396163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.396092 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "kube-api-access-w6cs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:34.396163 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.396116 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs" pod "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" (UID: "bda77944-7fb6-42fe-b0f7-a0d0be4f97ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:34.494851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.494821 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:34.494851 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.494850 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:34.495017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:34.494861 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cs8\" (UniqueName: \"kubernetes.io/projected/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea-kube-api-access-w6cs8\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:35.184597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.184554 5036 generic.go:356] "Generic (PLEG): container finished" podID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" containerID="cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd" exitCode=0 Nov 01 22:40:35.185116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.184606 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" Nov 01 22:40:35.185116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.184648 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" event={"ID":"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea","Type":"ContainerDied","Data":"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd"} Nov 01 22:40:35.185116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.184706 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm" event={"ID":"bda77944-7fb6-42fe-b0f7-a0d0be4f97ea","Type":"ContainerDied","Data":"177df8f53268e70172c6c1ce7f984e615de11dad5d215d3f9a69efc3acbed426"} Nov 01 22:40:35.185116 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.184732 5036 scope.go:122] "RemoveContainer" containerID="cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd" Nov 01 22:40:35.194901 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.194880 5036 scope.go:122] "RemoveContainer" containerID="cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd" Nov 01 22:40:35.195190 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:40:35.195172 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd\": container with ID starting with cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd not found: ID does not exist" containerID="cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd" Nov 01 22:40:35.195254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.195198 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd"} err="failed to get container status \"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd\": rpc error: code = NotFound desc = could not find container \"cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd\": container with ID starting with cb9f52632dcb3e798ab084b7316746ea99d009ac1cdbe636ef07c4e8706576fd not found: ID does not exist" Nov 01 22:40:35.205030 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.205004 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:40:35.209917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:35.209893 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-096081/simulator-single-main-096081-kserve-6b6c87c44-p9fwm"] Nov 01 22:40:36.444370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:36.444334 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" path="/var/lib/kubelet/pods/bda77944-7fb6-42fe-b0f7-a0d0be4f97ea/volumes" Nov 01 22:40:44.950745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:44.950705 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:40:48.858821 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:40:48.858778 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:40:49.586952 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:49.586914 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:40:55.075069 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.075028 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 22:40:55.075978 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.075253 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" podUID="22f49525-ea81-41ec-8c2b-8def80641301" containerName="manager" containerID="cri-o://6575cdb9f445056fd481e3b5671f51dc5c0564187cdcc759e91abb262e436400" gracePeriod=10 Nov 01 22:40:55.302391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.302359 5036 generic.go:356] "Generic (PLEG): container finished" podID="22f49525-ea81-41ec-8c2b-8def80641301" containerID="6575cdb9f445056fd481e3b5671f51dc5c0564187cdcc759e91abb262e436400" exitCode=0 Nov 01 22:40:55.302566 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.302434 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" event={"ID":"22f49525-ea81-41ec-8c2b-8def80641301","Type":"ContainerDied","Data":"6575cdb9f445056fd481e3b5671f51dc5c0564187cdcc759e91abb262e436400"} Nov 01 22:40:55.347176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.347149 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 22:40:55.418817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.418774 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:55.522433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.522399 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\") pod \"22f49525-ea81-41ec-8c2b-8def80641301\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " Nov 01 22:40:55.522633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.522457 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\" (UniqueName: \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\") pod \"22f49525-ea81-41ec-8c2b-8def80641301\" (UID: \"22f49525-ea81-41ec-8c2b-8def80641301\") " Nov 01 22:40:55.522887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.522859 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume" pod "22f49525-ea81-41ec-8c2b-8def80641301" (UID: "22f49525-ea81-41ec-8c2b-8def80641301"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:55.524608 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.524585 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7" pod "22f49525-ea81-41ec-8c2b-8def80641301" (UID: "22f49525-ea81-41ec-8c2b-8def80641301"). InnerVolumeSpecName "kube-api-access-fmnb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:55.624024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.623959 5036 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f49525-ea81-41ec-8c2b-8def80641301-extensions-socket-volume\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:55.624024 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:55.623983 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmnb7\" (UniqueName: \"kubernetes.io/projected/22f49525-ea81-41ec-8c2b-8def80641301-kube-api-access-fmnb7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:56.310331 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.310299 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" Nov 01 22:40:56.310832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.310335 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5" event={"ID":"22f49525-ea81-41ec-8c2b-8def80641301","Type":"ContainerDied","Data":"edbf016dc2aec5e3f9b09d527905ee609c67992a361660789e40c4e63dbc13f7"} Nov 01 22:40:56.310832 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.310397 5036 scope.go:122] "RemoveContainer" containerID="6575cdb9f445056fd481e3b5671f51dc5c0564187cdcc759e91abb262e436400" Nov 01 22:40:56.340152 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.340128 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 22:40:56.349306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.349279 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fhns5"] Nov 01 22:40:56.438150 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.438119 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f49525-ea81-41ec-8c2b-8def80641301" path="/var/lib/kubelet/pods/22f49525-ea81-41ec-8c2b-8def80641301/volumes" Nov 01 22:40:56.667745 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:40:56.667666 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:01.693579 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:01.693539 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:02.384171 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:02.384134 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:18.857067 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:41:18.857025 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:41:48.857429 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:41:48.857344 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:41:56.083754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.083717 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs"] Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084134 5036 cpu_manager.go:414] "containerMap: removing container" podUID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" containerName="main" Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084147 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" containerName="main" Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084173 5036 cpu_manager.go:414] "containerMap: removing container" podUID="22f49525-ea81-41ec-8c2b-8def80641301" containerName="manager" Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084179 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f49525-ea81-41ec-8c2b-8def80641301" containerName="manager" Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084242 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f49525-ea81-41ec-8c2b-8def80641301" containerName="manager" Nov 01 22:41:56.084272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.084253 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda77944-7fb6-42fe-b0f7-a0d0be4f97ea" containerName="main" Nov 01 22:41:56.086856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.086839 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.101233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.101214 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8grxv\"" Nov 01 22:41:56.112761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.112727 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs"] Nov 01 22:41:56.125691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.125666 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11a55c2e-6d5f-49c8-8482-811ac49cd25b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.125810 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.125705 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbb2\" (UniqueName: \"kubernetes.io/projected/11a55c2e-6d5f-49c8-8482-811ac49cd25b-kube-api-access-sxbb2\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.226518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.226473 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11a55c2e-6d5f-49c8-8482-811ac49cd25b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.226518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.226525 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbb2\" (UniqueName: \"kubernetes.io/projected/11a55c2e-6d5f-49c8-8482-811ac49cd25b-kube-api-access-sxbb2\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.226939 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.226918 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11a55c2e-6d5f-49c8-8482-811ac49cd25b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.244147 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.244126 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbb2\" (UniqueName: \"kubernetes.io/projected/11a55c2e-6d5f-49c8-8482-811ac49cd25b-kube-api-access-sxbb2\") pod \"kuadrant-operator-controller-manager-678d87766f-f4hrs\" (UID: \"11a55c2e-6d5f-49c8-8482-811ac49cd25b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.399270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.399206 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:56.682501 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:56.682472 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs"] Nov 01 22:41:56.684394 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:41:56.684364 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a55c2e_6d5f_49c8_8482_811ac49cd25b.slice/crio-89466b607a73e34cfc640d44a1c3ad3703c5427f05ecadd9feee04f7af30462c WatchSource:0}: Error finding container 89466b607a73e34cfc640d44a1c3ad3703c5427f05ecadd9feee04f7af30462c: Status 404 returned error can't find the container with id 89466b607a73e34cfc640d44a1c3ad3703c5427f05ecadd9feee04f7af30462c Nov 01 22:41:57.626573 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:57.626532 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" event={"ID":"11a55c2e-6d5f-49c8-8482-811ac49cd25b","Type":"ContainerStarted","Data":"9a12bcb1ef67f84f3541a7759dfd5ab70823e184b78c8fff532d794279b29fbd"} Nov 01 22:41:57.626993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:57.626593 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" event={"ID":"11a55c2e-6d5f-49c8-8482-811ac49cd25b","Type":"ContainerStarted","Data":"89466b607a73e34cfc640d44a1c3ad3703c5427f05ecadd9feee04f7af30462c"} Nov 01 22:41:57.626993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:57.626621 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:41:57.658643 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:41:57.658594 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" podStartSLOduration=1.6585800609999999 podStartE2EDuration="1.658580061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:57.656328087 +0000 UTC m=+3471.349308574" watchObservedRunningTime="2026-11-01 22:41:57.658580061 +0000 UTC m=+3471.351560546" Nov 01 22:42:01.746334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:01.746292 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:08.634666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:08.634633 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4hrs" Nov 01 22:42:13.443281 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:13.443235 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:42:18.856508 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:42:18.856468 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:42:28.290941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:28.290895 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:42:32.003649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:32.003611 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:42:47.191080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:47.191039 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:42:47.759289 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:47.759252 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.165076 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:48.165038 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.363278 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:48.363228 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.514295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:48.514195 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.857459 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:42:48.857396 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:42:51.523107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:51.523074 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:51.665784 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:51.665751 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:52.733890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:52.733852 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:52.875249 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:52.875209 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:53.008070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:53.007901 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:53.540194 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:53.540151 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:54.243987 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:54.243942 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:54.654405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:54.654366 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:42:55.139549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:55.139511 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:55.962259 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:55.962220 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:56.673472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:56.673408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:57.455167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:57.455131 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:58.162764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:58.162717 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:58.948957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:58.948922 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:59.760887 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:42:59.760836 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:00.557767 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:00.557731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.380715 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:01.380684 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.792075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:01.792042 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:02.230760 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:02.230665 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:02.931182 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:02.931141 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:03.636820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:03.636778 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:04.334504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:04.334471 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:05.140638 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:05.140605 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:05.847539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:05.847505 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:06.471602 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:06.471521 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:06.630260 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:06.630225 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.042598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.042555 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.180440 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.180405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.308098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.308006 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.625319 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.625218 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.968467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.968361 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:43:07.973067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.973041 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983093 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983061 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983157 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983195 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983261 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983255 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983280 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.983473 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.983322 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8rj\" (UniqueName: \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:07.987606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.987584 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3813a1\"/\"kube-root-ca.crt\"" Nov 01 22:43:07.987746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.987698 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3813a1\"/\"openshift-service-ca.crt\"" Nov 01 22:43:07.987830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.987815 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3813a1\"/\"default-dockercfg-6mg2d\"" Nov 01 22:43:07.987935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.987918 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3813a1\"/\"simulator-single-main-3813a1-kserve-self-signed-certs\"" Nov 01 22:43:07.988285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.988253 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:43:07.992894 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:07.992862 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:08.084301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084268 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.084794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084335 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.084794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084362 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.084794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084404 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.084794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084430 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.084794 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084482 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8rj\" (UniqueName: \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.085063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084954 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.085063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.084987 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.085063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.085020 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.086763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.086741 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.086882 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.086866 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.097427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.097403 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8rj\" (UniqueName: \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\") pod \"simulator-single-main-3813a1-kserve-7454dbfb75-57tpx\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.299436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.299399 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:08.566470 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.566356 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:08.586497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.586424 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:43:08.986317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:08.986224 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:43:09.036296 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:09.036258 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" event={"ID":"8f4ef5ab-6018-4c74-ad00-d55b91cc6825","Type":"ContainerStarted","Data":"941d898e9f2fae5f11517a34b2b1b4d683637028614489e69d841514676c3f72"} Nov 01 22:43:09.036520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:09.036307 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" event={"ID":"8f4ef5ab-6018-4c74-ad00-d55b91cc6825","Type":"ContainerStarted","Data":"00337f6a81cea055f0d1015b3a84fe845e981748a32fd61b2026145a680d410b"} Nov 01 22:43:09.060817 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:09.060760 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" podStartSLOduration=2.060746637 podStartE2EDuration="2.060746637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:09.05868857 +0000 UTC m=+3542.751669048" watchObservedRunningTime="2026-11-01 22:43:09.060746637 +0000 UTC m=+3542.753727123" Nov 01 22:43:09.300379 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:09.300342 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:09.427565 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:09.427436 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:10.289429 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:10.289392 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:11.052813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:11.052778 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:11.854776 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:11.854635 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:12.536332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:12.536296 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:13.341856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:13.341814 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:14.136616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:14.136576 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:14.906268 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:14.906230 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:15.559262 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:15.559224 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:16.219311 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:16.219273 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:16.923539 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:16.923490 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:17.636737 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:17.636647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:18.192263 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:18.192223 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:18.300603 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:18.300571 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:18.313251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:18.313227 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:18.316481 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:18.316426 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:18.458878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:18.458785 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:18.856748 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:43:18.856707 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:43:19.033666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:19.033632 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:19.101542 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:19.101493 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:43:19.845746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:19.845691 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:20.533809 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:20.533767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:21.235633 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:21.235600 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:21.946858 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:21.946812 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:22.645677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:22.645635 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:23.359324 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:23.358731 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:24.056056 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:24.056019 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:24.908407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:24.907117 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:25.636209 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:25.636165 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:25.893153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:25.893070 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:43:26.447094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:26.447057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:27.136499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:27.136463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:27.842597 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:27.842556 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:28.530842 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:28.530805 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:29.242618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:29.242580 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:29.940729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:29.940692 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:30.655992 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:30.655956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:31.355611 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:31.355572 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:32.061345 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:32.061308 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:32.752398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:32.752362 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:33.516314 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:33.516219 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:34.125677 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:34.125645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:34.932406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:34.932361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:35.643507 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:35.643468 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:36.340225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:36.340188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:37.041352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:37.041316 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:37.733657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:37.733616 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:38.447920 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:38.447882 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:39.246017 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:39.245976 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:40.153782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:40.153747 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:40.844864 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:40.844824 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:41.554175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:41.554132 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:42.245191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:42.245155 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:43.033090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:43.033055 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:43.732467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:43.732406 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:44.429827 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:44.429789 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:45.228394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:45.228350 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:46.029616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:46.029575 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:46.737928 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:46.737859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:47.529228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:47.529192 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:48.230812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:48.230773 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:48.857676 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:43:48.857635 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:43:48.933644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:48.933613 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:49.264763 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:49.264662 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:49.564649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:49.564615 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:43:49.738518 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:49.738481 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:50.524097 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:50.524064 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:51.230140 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:51.230102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:51.938131 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:51.938098 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:52.647099 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:52.647062 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:53.333387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:53.333352 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:54.033577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:54.033534 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:54.191098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:54.191060 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:43:54.304999 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:54.304920 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:54.776428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:54.776394 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:55.527133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:55.527097 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:56.228012 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:56.227972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:56.927199 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:56.927164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:57.630679 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:57.630630 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:58.433577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:58.433532 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.142181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:59.142143 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.581156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:59.581119 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.712688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:59.712647 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.842091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:43:59.841957 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:00.411204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:00.411164 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:00.544741 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:00.544695 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:00.945683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:00.945643 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:01.840599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:01.840551 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:01.891471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:01.891427 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:44:06.809409 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:06.809368 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:06.938354 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:06.938313 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:07.135946 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:07.135845 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:08.109997 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:08.109960 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:44:08.797989 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:08.797962 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:44:09.211184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.211084 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" podUID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" containerName="main" containerID="cri-o://941d898e9f2fae5f11517a34b2b1b4d683637028614489e69d841514676c3f72" gracePeriod=45 Nov 01 22:44:09.373336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.373292 5036 generic.go:356] "Generic (PLEG): container finished" podID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" containerID="941d898e9f2fae5f11517a34b2b1b4d683637028614489e69d841514676c3f72" exitCode=0 Nov 01 22:44:09.373522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.373360 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" event={"ID":"8f4ef5ab-6018-4c74-ad00-d55b91cc6825","Type":"ContainerDied","Data":"941d898e9f2fae5f11517a34b2b1b4d683637028614489e69d841514676c3f72"} Nov 01 22:44:09.461285 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.461228 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:44:09.583625 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583592 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.583804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583635 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.583804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583693 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.583804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583755 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.583804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583795 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\" (UniqueName: \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.584051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583860 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\") pod \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\" (UID: \"8f4ef5ab-6018-4c74-ad00-d55b91cc6825\") " Nov 01 22:44:09.584051 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.583906 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:09.584176 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.584054 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:09.584223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.584168 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:09.584279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.584260 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:09.584326 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.584284 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:09.586528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.586502 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:09.586528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.586508 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:09.586684 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.586575 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj" pod "8f4ef5ab-6018-4c74-ad00-d55b91cc6825" (UID: "8f4ef5ab-6018-4c74-ad00-d55b91cc6825"). InnerVolumeSpecName "kube-api-access-pp8rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:09.685080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.685049 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8rj\" (UniqueName: \"kubernetes.io/projected/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-kube-api-access-pp8rj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:09.685080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.685074 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:09.685080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.685084 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:09.685274 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:09.685092 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f4ef5ab-6018-4c74-ad00-d55b91cc6825-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:10.380993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.380956 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" event={"ID":"8f4ef5ab-6018-4c74-ad00-d55b91cc6825","Type":"ContainerDied","Data":"00337f6a81cea055f0d1015b3a84fe845e981748a32fd61b2026145a680d410b"} Nov 01 22:44:10.381580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.381008 5036 scope.go:122] "RemoveContainer" containerID="941d898e9f2fae5f11517a34b2b1b4d683637028614489e69d841514676c3f72" Nov 01 22:44:10.381580 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.380974 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx" Nov 01 22:44:10.406967 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.406941 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:44:10.413754 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.413730 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3813a1/simulator-single-main-3813a1-kserve-7454dbfb75-57tpx"] Nov 01 22:44:10.437664 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:10.437637 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" path="/var/lib/kubelet/pods/8f4ef5ab-6018-4c74-ad00-d55b91cc6825/volumes" Nov 01 22:44:12.827142 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.827106 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:44:12.827612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.827595 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" containerName="main" Nov 01 22:44:12.827665 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.827614 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" containerName="main" Nov 01 22:44:12.827709 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.827702 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4ef5ab-6018-4c74-ad00-d55b91cc6825" containerName="main" Nov 01 22:44:12.830915 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.830897 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:12.840865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.840839 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 22:44:12.841061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.840882 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rfgwk\"" Nov 01 22:44:12.841061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.840882 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 22:44:12.841061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.840962 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c0107984-kserve-self-signed-certs\"" Nov 01 22:44:12.844192 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.841927 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:44:12.916251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:12.916210 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:13.013424 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013385 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.013628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013477 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.013628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013592 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.013740 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013663 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwjj\" (UniqueName: \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.013826 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013801 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.013878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.013852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.114829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114733 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.114829 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114797 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114843 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114892 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114925 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.114974 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwjj\" (UniqueName: \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115380 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.115344 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.115504 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.115670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.115360 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.117641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.117615 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.117749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.117664 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.127637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.127612 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwjj\" (UniqueName: \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\") pod \"e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.158366 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.158338 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:13.461217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:13.461188 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:44:13.463060 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:44:13.463024 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9ab433_5752_4310_9b84_0b4fa236eb4a.slice/crio-95a866d62d0cd1c2428ff0c14fb3ae7fbeaccc59c6b8ff7d1dc7d5e59eea956f WatchSource:0}: Error finding container 95a866d62d0cd1c2428ff0c14fb3ae7fbeaccc59c6b8ff7d1dc7d5e59eea956f: Status 404 returned error can't find the container with id 95a866d62d0cd1c2428ff0c14fb3ae7fbeaccc59c6b8ff7d1dc7d5e59eea956f Nov 01 22:44:14.406695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:14.406653 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" event={"ID":"7f9ab433-5752-4310-9b84-0b4fa236eb4a","Type":"ContainerStarted","Data":"033f803b17ff398e96d68bd2498d8f1e3f2737c40c4a6178eafaacaddfaf916d"} Nov 01 22:44:14.406695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:14.406699 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" event={"ID":"7f9ab433-5752-4310-9b84-0b4fa236eb4a","Type":"ContainerStarted","Data":"95a866d62d0cd1c2428ff0c14fb3ae7fbeaccc59c6b8ff7d1dc7d5e59eea956f"} Nov 01 22:44:14.429503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:14.429440 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" podStartSLOduration=2.429427863 podStartE2EDuration="2.429427863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:14.428333177 +0000 UTC m=+3608.121313664" watchObservedRunningTime="2026-11-01 22:44:14.429427863 +0000 UTC m=+3608.122408340" Nov 01 22:44:14.457779 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:14.457742 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:44:15.159252 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:15.159212 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:17.194468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:17.193531 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:17.194468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:17.193605 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.857175 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:44:18.857137 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:44:23.158670 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:23.158616 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:23.171830 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:23.171801 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:23.500027 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:23.499947 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:44:27.142522 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.142483 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:44:27.148323 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.148300 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.154591 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.154559 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:44:27.179034 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179005 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.179189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179046 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.179254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179213 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.179339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179309 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqd9\" (UniqueName: \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.179509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179350 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.179509 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.179466 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.198886 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.198843 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c0107984-kserve-self-signed-certs\"" Nov 01 22:44:27.241717 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.241665 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:27.280250 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280208 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.280488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280274 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.280488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280345 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.280488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280389 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqd9\" (UniqueName: \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.280488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280418 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.280813 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280493 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.281218 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.280963 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.281976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.281676 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.281976 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.281913 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.285098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.284958 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.285699 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.285665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.296044 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.295167 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqd9\" (UniqueName: \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\") pod \"e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.465582 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.465497 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:27.746212 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:27.746175 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:44:27.748604 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:44:27.748570 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48bf1b4f_c892_4f58_9db6_48060bd9e099.slice/crio-a02dd233e35f3e8a0cb56a0d0fca0c23ed79d1fc60d23219d4f1a62c9d4749bc WatchSource:0}: Error finding container a02dd233e35f3e8a0cb56a0d0fca0c23ed79d1fc60d23219d4f1a62c9d4749bc: Status 404 returned error can't find the container with id a02dd233e35f3e8a0cb56a0d0fca0c23ed79d1fc60d23219d4f1a62c9d4749bc Nov 01 22:44:28.417883 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:28.417846 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:28.516222 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:28.516179 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" event={"ID":"48bf1b4f-c892-4f58-9db6-48060bd9e099","Type":"ContainerStarted","Data":"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb"} Nov 01 22:44:28.516421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:28.516231 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" event={"ID":"48bf1b4f-c892-4f58-9db6-48060bd9e099","Type":"ContainerStarted","Data":"a02dd233e35f3e8a0cb56a0d0fca0c23ed79d1fc60d23219d4f1a62c9d4749bc"} Nov 01 22:44:28.538010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:28.537921 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" podStartSLOduration=1.537895837 podStartE2EDuration="1.537895837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:28.535623961 +0000 UTC m=+3622.228604446" watchObservedRunningTime="2026-11-01 22:44:28.537895837 +0000 UTC m=+3622.230876325" Nov 01 22:44:28.968091 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:28.968043 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:44:29.466571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:29.466534 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:30.601466 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:30.600887 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:32.216469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:32.207542 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m9bdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:32.216469 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:32.207604 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m9bdz" podUID="fb61880c-597e-4979-bda6-9f24361aaabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:36.621899 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:36.621857 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:37.466406 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:37.466374 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:37.478959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:37.478929 5036 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:37.601587 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:37.601551 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:44:45.189749 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:45.189706 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:44:48.857286 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:44:48.857192 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:44:55.418520 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:55.418474 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:44:55.512804 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:55.512768 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:56.800063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:56.800014 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:44:56.822950 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:56.822915 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:44:56.876438 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:44:56.876402 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:00.139240 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.139195 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:00.148465 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.145908 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.157255 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.157230 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:00.158329 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.158310 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:45:00.158604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.158585 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:45:00.165468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.158762 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:45:00.184413 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.184363 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmph7\" (UniqueName: \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.184721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.184678 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.288751 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.288711 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmph7\" (UniqueName: \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.289369 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.289338 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.290617 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.290590 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.319953 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.319920 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmph7\" (UniqueName: \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\") pod \"maas-api-key-cleanup-29892885-rdfr5\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.489110 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.489011 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:00.772879 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:00.772842 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:01.775013 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:01.774970 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerStarted","Data":"0c00845dd159352899a0f7f4460ef390ce8bea1b72bd3c4f1eaba5e794ecc95e"} Nov 01 22:45:01.775548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:01.775059 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerStarted","Data":"77c354afc7bf63fdff5fe790c308674f150312c0d22ebe603dd3bfe5796b1d4e"} Nov 01 22:45:01.823462 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:01.819100 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" podStartSLOduration=1.819080573 podStartE2EDuration="1.819080573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:01.815800071 +0000 UTC m=+3655.508780558" watchObservedRunningTime="2026-11-01 22:45:01.819080573 +0000 UTC m=+3655.512061060" Nov 01 22:45:01.997467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:01.996986 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:02.424295 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:02.424257 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:45:02.802474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:02.801240 5036 generic.go:356] "Generic (PLEG): container finished" podID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerID="0c00845dd159352899a0f7f4460ef390ce8bea1b72bd3c4f1eaba5e794ecc95e" exitCode=60 Nov 01 22:45:02.802474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:02.801387 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerDied","Data":"0c00845dd159352899a0f7f4460ef390ce8bea1b72bd3c4f1eaba5e794ecc95e"} Nov 01 22:45:02.802474 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:02.801790 5036 scope.go:122] "RemoveContainer" containerID="0c00845dd159352899a0f7f4460ef390ce8bea1b72bd3c4f1eaba5e794ecc95e" Nov 01 22:45:03.808510 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:03.808479 5036 generic.go:356] "Generic (PLEG): container finished" podID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerID="a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922" exitCode=60 Nov 01 22:45:03.808957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:03.808546 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerDied","Data":"a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922"} Nov 01 22:45:03.808957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:03.808612 5036 scope.go:122] "RemoveContainer" containerID="0c00845dd159352899a0f7f4460ef390ce8bea1b72bd3c4f1eaba5e794ecc95e" Nov 01 22:45:03.808957 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:03.808848 5036 scope.go:122] "RemoveContainer" containerID="a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922" Nov 01 22:45:03.809138 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:03.809116 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-rdfr5_odh-ai-gateway-infra(9a4462dd-cf7c-497d-800e-eb02a43906cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" Nov 01 22:45:04.815651 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:04.815623 5036 scope.go:122] "RemoveContainer" containerID="a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922" Nov 01 22:45:04.816163 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:04.815828 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-rdfr5_odh-ai-gateway-infra(9a4462dd-cf7c-497d-800e-eb02a43906cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" Nov 01 22:45:09.284172 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:09.284128 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:45:10.466264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:10.466016 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerName="main" containerID="cri-o://033f803b17ff398e96d68bd2498d8f1e3f2737c40c4a6178eafaacaddfaf916d" gracePeriod=45 Nov 01 22:45:10.493929 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:10.489737 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerName="main" probeResult="failure" output="Get \"https://10.132.1.134:8000/ready\": dial tcp 10.132.1.134:8000: connect: connection refused" Nov 01 22:45:10.872270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:10.872237 5036 generic.go:356] "Generic (PLEG): container finished" podID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerID="033f803b17ff398e96d68bd2498d8f1e3f2737c40c4a6178eafaacaddfaf916d" exitCode=0 Nov 01 22:45:10.872423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:10.872284 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" event={"ID":"7f9ab433-5752-4310-9b84-0b4fa236eb4a","Type":"ContainerDied","Data":"033f803b17ff398e96d68bd2498d8f1e3f2737c40c4a6178eafaacaddfaf916d"} Nov 01 22:45:11.073714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.073589 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:45:11.275453 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275398 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.275650 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275564 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\" (UniqueName: \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.275734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275655 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.275734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275717 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.275865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275766 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.275865 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.275809 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\") pod \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\" (UID: \"7f9ab433-5752-4310-9b84-0b4fa236eb4a\") " Nov 01 22:45:11.283477 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.280833 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:11.283857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.283834 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:11.303569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.303536 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:11.315569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.313611 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:11.315868 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.315843 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:11.335648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.335607 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj" pod "7f9ab433-5752-4310-9b84-0b4fa236eb4a" (UID: "7f9ab433-5752-4310-9b84-0b4fa236eb4a"). InnerVolumeSpecName "kube-api-access-kbwjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377073 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377110 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377125 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377141 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwjj\" (UniqueName: \"kubernetes.io/projected/7f9ab433-5752-4310-9b84-0b4fa236eb4a-kube-api-access-kbwjj\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377159 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.377483 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.377173 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9ab433-5752-4310-9b84-0b4fa236eb4a-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:11.537064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.536628 5036 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tmn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:11.537064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.536710 5036 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tmn2k" podUID="90f246de-749e-4ada-81a3-765a18df9fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:11.850437 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.850333 5036 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" podUID="48bf1b4f-c892-4f58-9db6-48060bd9e099" containerName="main" containerID="cri-o://9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb" gracePeriod=45 Nov 01 22:45:11.878926 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.878898 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" Nov 01 22:45:11.879045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.878925 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5" event={"ID":"7f9ab433-5752-4310-9b84-0b4fa236eb4a","Type":"ContainerDied","Data":"95a866d62d0cd1c2428ff0c14fb3ae7fbeaccc59c6b8ff7d1dc7d5e59eea956f"} Nov 01 22:45:11.879045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.878979 5036 scope.go:122] "RemoveContainer" containerID="033f803b17ff398e96d68bd2498d8f1e3f2737c40c4a6178eafaacaddfaf916d" Nov 01 22:45:11.964426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.964399 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:45:11.967739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:11.967713 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c0107984-kserve-76f8bf666f-bftl5"] Nov 01 22:45:12.150000 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.149967 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185269 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185321 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185375 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\" (UniqueName: \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185417 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185518 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.185583 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\" (UniqueName: \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\") pod \"48bf1b4f-c892-4f58-9db6-48060bd9e099\" (UID: \"48bf1b4f-c892-4f58-9db6-48060bd9e099\") " Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.186478 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.186547 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:12.190351 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.186753 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:12.191683 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.191651 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "kube-api-access-ptqd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:12.193114 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.193076 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:12.208146 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.193530 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs" pod "48bf1b4f-c892-4f58-9db6-48060bd9e099" (UID: "48bf1b4f-c892-4f58-9db6-48060bd9e099"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:12.286944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.286898 5036 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-dshm\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.286944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.286948 5036 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-home\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.287208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.286965 5036 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48bf1b4f-c892-4f58-9db6-48060bd9e099-tls-certs\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.287208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.286981 5036 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-model-cache\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.287208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.286997 5036 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48bf1b4f-c892-4f58-9db6-48060bd9e099-tmp-dir\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.287208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.287014 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqd9\" (UniqueName: \"kubernetes.io/projected/48bf1b4f-c892-4f58-9db6-48060bd9e099-kube-api-access-ptqd9\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:12.440623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.440533 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" path="/var/lib/kubelet/pods/7f9ab433-5752-4310-9b84-0b4fa236eb4a/volumes" Nov 01 22:45:12.892969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.889234 5036 generic.go:356] "Generic (PLEG): container finished" podID="48bf1b4f-c892-4f58-9db6-48060bd9e099" containerID="9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb" exitCode=0 Nov 01 22:45:12.892969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.889335 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" Nov 01 22:45:12.892969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.889354 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" event={"ID":"48bf1b4f-c892-4f58-9db6-48060bd9e099","Type":"ContainerDied","Data":"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb"} Nov 01 22:45:12.892969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.889410 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw" event={"ID":"48bf1b4f-c892-4f58-9db6-48060bd9e099","Type":"ContainerDied","Data":"a02dd233e35f3e8a0cb56a0d0fca0c23ed79d1fc60d23219d4f1a62c9d4749bc"} Nov 01 22:45:12.892969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.889437 5036 scope.go:122] "RemoveContainer" containerID="9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb" Nov 01 22:45:12.901216 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.901192 5036 scope.go:122] "RemoveContainer" containerID="9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb" Nov 01 22:45:12.901677 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:12.901653 5036 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb\": container with ID starting with 9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb not found: ID does not exist" containerID="9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb" Nov 01 22:45:12.901783 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.901688 5036 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb"} err="failed to get container status \"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb\": rpc error: code = NotFound desc = could not find container \"9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb\": container with ID starting with 9ec3436f20d030cc04a82c72131b58089e7222e86adfc9a3adfc27a736a069bb not found: ID does not exist" Nov 01 22:45:12.910421 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.910387 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:45:12.912839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:12.912812 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c0107984-kserve-7c5979fffc-wrvcw"] Nov 01 22:45:13.653554 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:13.653511 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:45:14.452812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:14.452757 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bf1b4f-c892-4f58-9db6-48060bd9e099" path="/var/lib/kubelet/pods/48bf1b4f-c892-4f58-9db6-48060bd9e099/volumes" Nov 01 22:45:15.430972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:15.430941 5036 scope.go:122] "RemoveContainer" containerID="a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922" Nov 01 22:45:15.922605 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:15.922564 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerStarted","Data":"1a9b1da3ea446a8b883b8621e9425379a26e215061a168b66a9d36302d42ac2b"} Nov 01 22:45:16.930181 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:16.930147 5036 generic.go:356] "Generic (PLEG): container finished" podID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerID="1a9b1da3ea446a8b883b8621e9425379a26e215061a168b66a9d36302d42ac2b" exitCode=60 Nov 01 22:45:16.930694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:16.930225 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerDied","Data":"1a9b1da3ea446a8b883b8621e9425379a26e215061a168b66a9d36302d42ac2b"} Nov 01 22:45:16.930694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:16.930290 5036 scope.go:122] "RemoveContainer" containerID="a19a21007e69fd4edc756e0b43b3480c05a0e85f499d1280d3ad08480d740922" Nov 01 22:45:16.930694 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:16.930672 5036 scope.go:122] "RemoveContainer" containerID="1a9b1da3ea446a8b883b8621e9425379a26e215061a168b66a9d36302d42ac2b" Nov 01 22:45:16.930976 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:16.930954 5036 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-rdfr5_odh-ai-gateway-infra(9a4462dd-cf7c-497d-800e-eb02a43906cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" Nov 01 22:45:16.957854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:16.957824 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:18.071107 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.071080 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:18.176088 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.176056 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\" (UniqueName: \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\") pod \"9a4462dd-cf7c-497d-800e-eb02a43906cf\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " Nov 01 22:45:18.176297 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.176158 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\") pod \"9a4462dd-cf7c-497d-800e-eb02a43906cf\" (UID: \"9a4462dd-cf7c-497d-800e-eb02a43906cf\") " Nov 01 22:45:18.176647 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.176611 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca" pod "9a4462dd-cf7c-497d-800e-eb02a43906cf" (UID: "9a4462dd-cf7c-497d-800e-eb02a43906cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:18.178227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.178200 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7" pod "9a4462dd-cf7c-497d-800e-eb02a43906cf" (UID: "9a4462dd-cf7c-497d-800e-eb02a43906cf"). InnerVolumeSpecName "kube-api-access-hmph7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:18.277856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.277824 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmph7\" (UniqueName: \"kubernetes.io/projected/9a4462dd-cf7c-497d-800e-eb02a43906cf-kube-api-access-hmph7\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:18.277856 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.277854 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4462dd-cf7c-497d-800e-eb02a43906cf-maas-api-service-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:18.856662 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:18.856620 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:45:18.943204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.943178 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" Nov 01 22:45:18.943387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.943175 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5" event={"ID":"9a4462dd-cf7c-497d-800e-eb02a43906cf","Type":"ContainerDied","Data":"77c354afc7bf63fdff5fe790c308674f150312c0d22ebe603dd3bfe5796b1d4e"} Nov 01 22:45:18.943387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.943291 5036 scope.go:122] "RemoveContainer" containerID="1a9b1da3ea446a8b883b8621e9425379a26e215061a168b66a9d36302d42ac2b" Nov 01 22:45:18.962839 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.962813 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:18.968635 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:18.968615 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-rdfr5"] Nov 01 22:45:19.960631 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:19.960535 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:20.436857 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:20.436819 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" path="/var/lib/kubelet/pods/9a4462dd-cf7c-497d-800e-eb02a43906cf/volumes" Nov 01 22:45:29.366807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:29.366765 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:45:37.382685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:37.382645 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:37.858404 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:37.858356 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:38.062667 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:38.062604 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:40.091023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:40.090972 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:40.233143 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:40.233108 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:40.532534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:40.532501 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.373593 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:41.373561 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.506377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:41.506340 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.640653 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:41.640565 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.760398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:41.760364 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:42.443246 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:42.443205 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:43.145881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:43.145834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:43.585066 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:43.585029 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:45:43.932301 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:43.932190 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:44.754917 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:44.754879 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:45.446367 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:45.446332 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:46.152998 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:46.152958 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:46.850385 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:46.850352 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:47.554621 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:47.554584 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.248050 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:48.248015 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.858592 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:48.858554 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:45:48.961328 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:48.961293 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:49.698598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:49.698559 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:50.451210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:50.451168 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:51.197183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:51.197140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:51.843178 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:51.843135 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:52.530398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:52.530360 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:53.237173 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:53.237138 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:54.037394 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:54.037361 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:54.748317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:54.748278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:55.615183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.615145 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:55.785153 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785119 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv"] Nov 01 22:45:55.785571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785559 5036 cpu_manager.go:414] "containerMap: removing container" podUID="48bf1b4f-c892-4f58-9db6-48060bd9e099" containerName="main" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785573 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bf1b4f-c892-4f58-9db6-48060bd9e099" containerName="main" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785586 5036 cpu_manager.go:414] "containerMap: removing container" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerName="main" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785591 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerName="main" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785605 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785611 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785626 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785630 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785637 5036 cpu_manager.go:414] "containerMap: removing container" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785642 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785697 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9ab433-5752-4310-9b84-0b4fa236eb4a" containerName="main" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785705 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785713 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:45:55.785849 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.785723 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bf1b4f-c892-4f58-9db6-48060bd9e099" containerName="main" Nov 01 22:45:55.789001 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.788983 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:55.795276 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.795249 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:45:55.796884 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.796865 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:45:55.797408 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.797256 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:55.797732 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.797715 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:45:55.799746 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.799723 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv"] Nov 01 22:45:55.910251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.910177 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:55.910251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.910249 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:55.910434 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:55.910363 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.011279 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.011246 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.011480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.011308 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.011480 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.011357 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.012072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.012051 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.013686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.013665 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.017613 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:56.017590 5036 projected.go:196] Error preparing data for projected volume kube-api-access-c2t4f for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 01 22:45:56.017731 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:45:56.017647 5036 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f podName:645540d1-f128-49c1-97c2-87194d29c7c1 nodeName:}" failed. No retries permitted until 2026-11-01 22:45:56.517633025 +0000 UTC m=+3710.210613489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c2t4f" (UniqueName: "kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f") pod "maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" (UID: "645540d1-f128-49c1-97c2-87194d29c7c1") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 01 22:45:56.457571 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.457527 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:56.617974 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.617937 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.630604 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.630576 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") pod \"maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.704533 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.704504 5036 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" secret="" err="secret \"maas-api-cleanup-dockercfg-mvwcj\" not found" Nov 01 22:45:56.704685 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.704557 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:56.981221 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:56.981190 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv"] Nov 01 22:45:56.982917 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:45:56.982888 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645540d1_f128_49c1_97c2_87194d29c7c1.slice/crio-36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563 WatchSource:0}: Error finding container 36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563: Status 404 returned error can't find the container with id 36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563 Nov 01 22:45:57.130802 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:57.130768 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:57.158524 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:57.158484 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" event={"ID":"645540d1-f128-49c1-97c2-87194d29c7c1","Type":"ContainerStarted","Data":"36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563"} Nov 01 22:45:57.684833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:57.684749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:57.844238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:57.844194 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:58.165155 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:58.165125 5036 generic.go:356] "Generic (PLEG): container finished" podID="645540d1-f128-49c1-97c2-87194d29c7c1" containerID="6606298ffe273110b39c25a5721dff7304f57d3b8e259c0077e39a91c0d00a73" exitCode=0 Nov 01 22:45:58.165558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:58.165223 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" event={"ID":"645540d1-f128-49c1-97c2-87194d29c7c1","Type":"ContainerDied","Data":"6606298ffe273110b39c25a5721dff7304f57d3b8e259c0077e39a91c0d00a73"} Nov 01 22:45:58.543937 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:58.543901 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:59.250648 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.250612 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:59.310764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.310736 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:45:59.445352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.445276 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") pod \"645540d1-f128-49c1-97c2-87194d29c7c1\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " Nov 01 22:45:59.445352 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.445331 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\") pod \"645540d1-f128-49c1-97c2-87194d29c7c1\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " Nov 01 22:45:59.445627 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.445391 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\") pod \"645540d1-f128-49c1-97c2-87194d29c7c1\" (UID: \"645540d1-f128-49c1-97c2-87194d29c7c1\") " Nov 01 22:45:59.445819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.445787 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca" pod "645540d1-f128-49c1-97c2-87194d29c7c1" (UID: "645540d1-f128-49c1-97c2-87194d29c7c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:59.447504 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.447482 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca" pod "645540d1-f128-49c1-97c2-87194d29c7c1" (UID: "645540d1-f128-49c1-97c2-87194d29c7c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:59.447642 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.447627 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f" pod "645540d1-f128-49c1-97c2-87194d29c7c1" (UID: "645540d1-f128-49c1-97c2-87194d29c7c1"). InnerVolumeSpecName "kube-api-access-c2t4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.546058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.546023 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.546058 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.546060 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2t4f\" (UniqueName: \"kubernetes.io/projected/645540d1-f128-49c1-97c2-87194d29c7c1-kube-api-access-c2t4f\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.546247 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.546076 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645540d1-f128-49c1-97c2-87194d29c7c1-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.934959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:45:59.934906 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:00.177657 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:00.177626 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" Nov 01 22:46:00.177824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:00.177619 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv" event={"ID":"645540d1-f128-49c1-97c2-87194d29c7c1","Type":"ContainerDied","Data":"36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563"} Nov 01 22:46:00.177824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:00.177755 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ab96a21db7e4a2d8bd7f508ccc06187bc2be86558e0cd3b7157f551263f563" Nov 01 22:46:00.638599 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:00.638563 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:01.322616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:01.322585 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:02.036675 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:02.036637 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:02.169179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:02.169137 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:02.733427 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:02.733380 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:03.554726 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:03.554679 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:04.235761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:04.235721 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:05.103439 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:05.103398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:05.744918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:05.744881 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.027863 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.027828 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.147422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.147389 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.342022 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.341928 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.494234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.494196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.563423 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.563390 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv"] Nov 01 22:46:06.566067 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.566035 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eaf797bf-pqgpv"] Nov 01 22:46:06.614961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:06.614886 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:07.843130 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:07.843086 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:46:08.437280 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:08.437243 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645540d1-f128-49c1-97c2-87194d29c7c1" path="/var/lib/kubelet/pods/645540d1-f128-49c1-97c2-87194d29c7c1/volumes" Nov 01 22:46:12.159101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:12.159057 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.384515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:13.384473 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.797387 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:13.797348 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.937253 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:13.937205 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:16.889005 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:16.888962 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:17.015918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:17.015878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.102497 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:18.102373 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.240433 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:18.240398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.367198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:18.367104 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.857078 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:46:18.857045 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:46:18.959801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:18.959763 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:19.767339 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:19.767296 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:20.058179 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:20.058145 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:46:20.628796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:20.628758 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.439612 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:21.439575 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.637436 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:21.637398 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.756357 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:21.756248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:22.194598 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:22.194554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:22.832705 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:22.832656 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:23.577891 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:23.577850 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:24.240979 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:24.240943 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:25.039555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:25.039518 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:25.735063 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:25.735024 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:26.502661 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:26.502617 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:27.250549 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:27.250505 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:27.942903 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:27.942866 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:28.664006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:28.663969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:29.351288 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:29.351248 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:30.042801 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:30.042767 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:30.754185 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:30.754140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:31.524106 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:31.524065 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:31.844959 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:31.844869 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:32.349072 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:32.349031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:33.046729 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:33.046686 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:33.843569 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:33.843083 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:34.540160 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:34.540125 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:35.241004 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:35.240969 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:35.951985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:35.951947 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:36.753624 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:36.753592 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:37.446241 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:37.446206 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:38.189855 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:38.189815 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:38.846993 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:38.846956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:39.616312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:39.616278 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:40.329854 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:40.329819 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:41.046734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:41.046694 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:41.764956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:41.764915 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:42.433687 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:42.433624 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.235133 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:43.235099 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.943632 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:43.943597 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:44.631251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:44.631210 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:45.353315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:45.353276 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:46.049121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:46.049078 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:46.832867 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:46.832834 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:47.533136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:47.533095 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:48.236836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:48.236790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:48.856363 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:46:48.856329 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:46:49.042995 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:49.042961 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.761080 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:49.761037 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:50.429203 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:50.429167 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:51.140085 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:51.140049 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:51.838793 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:51.838749 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:52.525365 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:52.525323 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:53.322835 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:53.322791 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:54.034079 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:54.034039 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:54.746935 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:54.746902 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:55.429390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:55.429357 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:56.139102 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:56.139056 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:56.837426 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:56.837385 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:57.538219 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:57.538180 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:58.239032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:58.238997 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:58.947911 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:58.947871 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:59.642254 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:46:59.642216 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:00.415191 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:00.415153 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:01.043972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:01.043923 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:01.831299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:01.831260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:02.089229 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:02.089142 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:02.222320 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:02.222285 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:02.534889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:02.534859 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:03.237175 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:03.237140 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:03.940407 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:03.940371 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:04.655962 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:04.655911 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:05.401277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:05.401229 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:06.049714 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:06.049678 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:06.721453 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:06.721405 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.439543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:07.439505 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:08.143117 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:08.143073 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:08.911036 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:08.910996 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:09.648576 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:09.648537 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:10.340505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:10.340471 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:11.045649 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:11.045614 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:11.863390 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:11.863358 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:12.648138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:12.648088 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:13.346803 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:13.346762 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:14.032217 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:14.032180 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:14.740031 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:14.739991 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:15.449601 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:15.449567 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:16.155098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:16.155060 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:16.843251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:16.843215 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:17.553782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:17.553737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:18.234674 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:18.234640 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:18.857836 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:47:18.857788 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:47:18.938681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:18.938648 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:19.639109 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:19.639071 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:20.345052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:20.345017 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:21.135588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:21.135543 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:21.832139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:21.832094 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:22.540210 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:22.540165 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:23.235618 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:23.235578 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:23.966708 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:23.966670 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:24.707334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:24.707302 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:25.429521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:25.429463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:26.177636 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:26.177603 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:26.988590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:26.988554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:27.731720 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:27.731675 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:28.456944 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:28.456891 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:29.149070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:29.149031 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:29.837304 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:29.837266 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:30.536234 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:30.536195 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:31.242198 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:31.242162 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:32.049721 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:32.049685 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:32.766251 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:32.766215 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:33.450590 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:33.450554 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.138370 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:34.138332 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.854486 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:34.854421 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.545860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:35.545819 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:36.259833 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:36.259790 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:36.937355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:36.937314 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:37.739969 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:37.739935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:38.479656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:38.479621 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:39.139945 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:39.139907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:39.952491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:39.952440 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:40.744457 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:40.744378 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:41.439731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:41.439699 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:42.195453 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:42.195409 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:42.937312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:42.937273 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:43.635966 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:43.635905 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:44.355037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:44.354998 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:45.031208 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:45.031172 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:45.738075 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:45.738027 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:46.442098 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:46.442059 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:47.224223 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:47.224175 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:47.951226 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:47.951188 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:48.644822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:48.644782 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:48.856497 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:47:48.856456 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:47:49.351053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:49.351016 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:50.052778 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:50.052737 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:50.746228 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:50.746196 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:51.445225 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.445185 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:51.490189 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.490153 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8"] Nov 01 22:47:51.490628 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.490610 5036 cpu_manager.go:414] "containerMap: removing container" podUID="645540d1-f128-49c1-97c2-87194d29c7c1" containerName="revoke-keys" Nov 01 22:47:51.490691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.490629 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645540d1-f128-49c1-97c2-87194d29c7c1" containerName="revoke-keys" Nov 01 22:47:51.490731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.490710 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4462dd-cf7c-497d-800e-eb02a43906cf" containerName="cleanup" Nov 01 22:47:51.490731 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.490722 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="645540d1-f128-49c1-97c2-87194d29c7c1" containerName="revoke-keys" Nov 01 22:47:51.493936 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.493914 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.501204 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.501171 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:47:51.502499 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.502482 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:47:51.502577 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.502531 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz99t\"" Nov 01 22:47:51.502792 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.502773 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:47:51.506558 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.506533 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8"] Nov 01 22:47:51.640298 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.640268 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.640476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.640304 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.640476 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.640326 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcgb\" (UniqueName: \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.741201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.741125 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.741201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.741164 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.741201 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.741184 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcgb\" (UniqueName: \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.741836 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.741811 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.743758 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.743732 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.753471 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.753429 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcgb\" (UniqueName: \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:51.806975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:51.806934 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:52.082047 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:52.082023 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8"] Nov 01 22:47:52.083829 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:47:52.083799 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a309e4_1d7b_48be_83b5_c73fca105b76.slice/crio-5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a WatchSource:0}: Error finding container 5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a: Status 404 returned error can't find the container with id 5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a Nov 01 22:47:52.326615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:52.326586 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:52.796853 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:52.796816 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" event={"ID":"91a309e4-1d7b-48be-83b5-c73fca105b76","Type":"ContainerStarted","Data":"c29f64e857e8d209bccd11bdf8410f47abdd9525385ca48a03f7fb2cc3bca593"} Nov 01 22:47:52.797264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:52.796877 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" event={"ID":"91a309e4-1d7b-48be-83b5-c73fca105b76","Type":"ContainerStarted","Data":"5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a"} Nov 01 22:47:52.817609 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:52.817559 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" podStartSLOduration=1.817539059 podStartE2EDuration="1.817539059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:52.816210158 +0000 UTC m=+3826.509190645" watchObservedRunningTime="2026-11-01 22:47:52.817539059 +0000 UTC m=+3826.510519545" Nov 01 22:47:53.038871 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:53.038838 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:53.751237 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:53.751199 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:53.803932 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:53.803898 5036 generic.go:356] "Generic (PLEG): container finished" podID="91a309e4-1d7b-48be-83b5-c73fca105b76" containerID="c29f64e857e8d209bccd11bdf8410f47abdd9525385ca48a03f7fb2cc3bca593" exitCode=0 Nov 01 22:47:53.804343 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:53.803986 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" event={"ID":"91a309e4-1d7b-48be-83b5-c73fca105b76","Type":"ContainerDied","Data":"c29f64e857e8d209bccd11bdf8410f47abdd9525385ca48a03f7fb2cc3bca593"} Nov 01 22:47:54.438910 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:54.438878 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:54.946090 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:54.946064 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:55.072860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.072776 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\") pod \"91a309e4-1d7b-48be-83b5-c73fca105b76\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " Nov 01 22:47:55.072860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.072820 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\") pod \"91a309e4-1d7b-48be-83b5-c73fca105b76\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " Nov 01 22:47:55.072860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.072844 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\" (UniqueName: \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\") pod \"91a309e4-1d7b-48be-83b5-c73fca105b76\" (UID: \"91a309e4-1d7b-48be-83b5-c73fca105b76\") " Nov 01 22:47:55.073270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.073244 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca" pod "91a309e4-1d7b-48be-83b5-c73fca105b76" (UID: "91a309e4-1d7b-48be-83b5-c73fca105b76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:55.075053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.075023 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb" pod "91a309e4-1d7b-48be-83b5-c73fca105b76" (UID: "91a309e4-1d7b-48be-83b5-c73fca105b76"). InnerVolumeSpecName "kube-api-access-xpcgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:55.075167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.075082 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca" pod "91a309e4-1d7b-48be-83b5-c73fca105b76" (UID: "91a309e4-1d7b-48be-83b5-c73fca105b76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:55.146965 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.146935 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:55.174401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.174366 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:55.174401 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.174398 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a309e4-1d7b-48be-83b5-c73fca105b76-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:55.174637 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.174410 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcgb\" (UniqueName: \"kubernetes.io/projected/91a309e4-1d7b-48be-83b5-c73fca105b76-kube-api-access-xpcgb\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:55.815374 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.815341 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" Nov 01 22:47:55.815588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.815341 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8" event={"ID":"91a309e4-1d7b-48be-83b5-c73fca105b76","Type":"ContainerDied","Data":"5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a"} Nov 01 22:47:55.815588 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.815486 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2f5171cfaa6e196c7dae31e86ddc15245265961f4a310c28740ae4a063be9a" Nov 01 22:47:55.855460 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:55.855408 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:56.553765 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:56.553707 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:57.250137 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:57.250105 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:57.998956 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:57.998918 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:58.743769 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:58.743734 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:59.588761 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:47:59.588728 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:00.350503 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:00.350463 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:01.059943 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:01.059907 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:01.763564 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:01.763522 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:01.939492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:01.939454 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:02.202158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:02.202106 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.595750 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:07.595712 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.740023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:07.739982 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.740764 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:07.740710 5036 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8"] Nov 01 22:48:07.743543 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:07.743516 5036 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1fef08-sq7q8"] Nov 01 22:48:07.874122 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:07.874021 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:08.283302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:08.283264 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:08.437287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:08.437253 5036 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a309e4-1d7b-48be-83b5-c73fca105b76" path="/var/lib/kubelet/pods/91a309e4-1d7b-48be-83b5-c73fca105b76/volumes" Nov 01 22:48:09.145866 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:09.145831 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:48:14.120184 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:14.120144 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:14.247305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:14.247260 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:14.456494 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:14.456375 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:16.289893 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:16.289854 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:48:18.875550 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:48:18.875517 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:48:21.392053 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:21.392015 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:48:25.639947 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.639911 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79"] Nov 01 22:48:25.640336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.640296 5036 cpu_manager.go:414] "containerMap: removing container" podUID="91a309e4-1d7b-48be-83b5-c73fca105b76" containerName="revoke-keys" Nov 01 22:48:25.640336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.640307 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a309e4-1d7b-48be-83b5-c73fca105b76" containerName="revoke-keys" Nov 01 22:48:25.640422 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.640413 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a309e4-1d7b-48be-83b5-c73fca105b76" containerName="revoke-keys" Nov 01 22:48:25.644782 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.644761 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.649924 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.649899 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:25.652528 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.651423 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz99t\"" Nov 01 22:48:25.654272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.654248 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79"] Nov 01 22:48:25.684881 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.684852 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.684985 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.684887 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jw62\" (UniqueName: \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.685032 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.685006 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.706961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.706934 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:25.706961 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.706954 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:25.785695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.785663 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.785822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.785707 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jw62\" (UniqueName: \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.785822 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.785796 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.786305 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.786288 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.788136 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.788119 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.799188 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.799165 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jw62\" (UniqueName: \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\") pod \"maas-api-revoke-sub-f03629db-m5c79\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:25.960165 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:25.960101 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:26.235572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:26.235546 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79"] Nov 01 22:48:26.237695 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:48:26.237660 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15be0371_1c7c_4cda_b948_3ba4c23cda59.slice/crio-b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314 WatchSource:0}: Error finding container b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314: Status 404 returned error can't find the container with id b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314 Nov 01 22:48:27.041006 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:27.040976 5036 generic.go:356] "Generic (PLEG): container finished" podID="15be0371-1c7c-4cda-b948-3ba4c23cda59" containerID="65e277eb8ccc30529af5f704b9edc28660b98fb01ee35d44571226ed64433039" exitCode=0 Nov 01 22:48:27.041361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:27.041009 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" event={"ID":"15be0371-1c7c-4cda-b948-3ba4c23cda59","Type":"ContainerDied","Data":"65e277eb8ccc30529af5f704b9edc28660b98fb01ee35d44571226ed64433039"} Nov 01 22:48:27.041361 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:27.041073 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" event={"ID":"15be0371-1c7c-4cda-b948-3ba4c23cda59","Type":"ContainerStarted","Data":"b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314"} Nov 01 22:48:28.180748 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.180722 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:28.207105 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.207079 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\" (UniqueName: \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\") pod \"15be0371-1c7c-4cda-b948-3ba4c23cda59\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " Nov 01 22:48:28.207299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.207154 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\") pod \"15be0371-1c7c-4cda-b948-3ba4c23cda59\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " Nov 01 22:48:28.207299 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.207240 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\") pod \"15be0371-1c7c-4cda-b948-3ba4c23cda59\" (UID: \"15be0371-1c7c-4cda-b948-3ba4c23cda59\") " Nov 01 22:48:28.207728 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.207695 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca" pod "15be0371-1c7c-4cda-b948-3ba4c23cda59" (UID: "15be0371-1c7c-4cda-b948-3ba4c23cda59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:28.209303 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.209274 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca" pod "15be0371-1c7c-4cda-b948-3ba4c23cda59" (UID: "15be0371-1c7c-4cda-b948-3ba4c23cda59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:28.209405 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.209345 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62" pod "15be0371-1c7c-4cda-b948-3ba4c23cda59" (UID: "15be0371-1c7c-4cda-b948-3ba4c23cda59"). InnerVolumeSpecName "kube-api-access-2jw62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:28.308606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.308529 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jw62\" (UniqueName: \"kubernetes.io/projected/15be0371-1c7c-4cda-b948-3ba4c23cda59-kube-api-access-2jw62\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:28.308606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.308560 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:28.308606 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:28.308576 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15be0371-1c7c-4cda-b948-3ba4c23cda59-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:29.052112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:29.052075 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" Nov 01 22:48:29.052112 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:29.052096 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m5c79" event={"ID":"15be0371-1c7c-4cda-b948-3ba4c23cda59","Type":"ContainerDied","Data":"b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314"} Nov 01 22:48:29.052334 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:29.052144 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20738be889711b26ac0a9500e594d9b29bbc4f5d250b520a8efcba86ebc8314" Nov 01 22:48:36.462242 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:36.462204 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:48:41.537919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.537875 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v"] Nov 01 22:48:41.538472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.538282 5036 cpu_manager.go:414] "containerMap: removing container" podUID="15be0371-1c7c-4cda-b948-3ba4c23cda59" containerName="revoke-keys" Nov 01 22:48:41.538472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.538294 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15be0371-1c7c-4cda-b948-3ba4c23cda59" containerName="revoke-keys" Nov 01 22:48:41.538472 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.538379 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="15be0371-1c7c-4cda-b948-3ba4c23cda59" containerName="revoke-keys" Nov 01 22:48:41.541820 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.541798 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.548045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.547944 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:41.548045 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.548012 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:41.550277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.550081 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:41.550277 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.550188 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz99t\"" Nov 01 22:48:41.554126 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.554097 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v"] Nov 01 22:48:41.660930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.660895 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.660930 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.660937 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gxf\" (UniqueName: \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.661183 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.660956 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.761488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.761460 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.761696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.761494 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gxf\" (UniqueName: \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.761696 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.761608 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.762238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.762217 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.763828 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.763801 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.773623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.773592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gxf\" (UniqueName: \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\") pod \"maas-api-revoke-sub-0165d7d2-csn7v\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:41.856644 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:41.856579 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:42.131691 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:42.131664 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v"] Nov 01 22:48:42.133986 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:48:42.133953 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b65bf70_4806_4ed8_b354_614fcbdd0403.slice/crio-cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c WatchSource:0}: Error finding container cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c: Status 404 returned error can't find the container with id cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c Nov 01 22:48:43.139492 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:43.139399 5036 generic.go:356] "Generic (PLEG): container finished" podID="8b65bf70-4806-4ed8-b354-614fcbdd0403" containerID="d0e9252748df33ac466f62dcb1433837f86dbf29de4a9df03ea0f9ea7c23b5dc" exitCode=0 Nov 01 22:48:43.139890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:43.139482 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" event={"ID":"8b65bf70-4806-4ed8-b354-614fcbdd0403","Type":"ContainerDied","Data":"d0e9252748df33ac466f62dcb1433837f86dbf29de4a9df03ea0f9ea7c23b5dc"} Nov 01 22:48:43.139890 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:43.139545 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" event={"ID":"8b65bf70-4806-4ed8-b354-614fcbdd0403","Type":"ContainerStarted","Data":"cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c"} Nov 01 22:48:44.281391 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.281361 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:44.384572 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.384532 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\") pod \"8b65bf70-4806-4ed8-b354-614fcbdd0403\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " Nov 01 22:48:44.384756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.384673 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\") pod \"8b65bf70-4806-4ed8-b354-614fcbdd0403\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " Nov 01 22:48:44.384756 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.384722 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\" (UniqueName: \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\") pod \"8b65bf70-4806-4ed8-b354-614fcbdd0403\" (UID: \"8b65bf70-4806-4ed8-b354-614fcbdd0403\") " Nov 01 22:48:44.385025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.385002 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca" pod "8b65bf70-4806-4ed8-b354-614fcbdd0403" (UID: "8b65bf70-4806-4ed8-b354-614fcbdd0403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:44.386645 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.386625 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca" pod "8b65bf70-4806-4ed8-b354-614fcbdd0403" (UID: "8b65bf70-4806-4ed8-b354-614fcbdd0403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:44.386824 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.386801 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf" pod "8b65bf70-4806-4ed8-b354-614fcbdd0403" (UID: "8b65bf70-4806-4ed8-b354-614fcbdd0403"). InnerVolumeSpecName "kube-api-access-45gxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:44.485898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.485871 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:44.485898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.485896 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gxf\" (UniqueName: \"kubernetes.io/projected/8b65bf70-4806-4ed8-b354-614fcbdd0403-kube-api-access-45gxf\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:44.486103 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:44.485908 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b65bf70-4806-4ed8-b354-614fcbdd0403-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:45.150927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:45.150894 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" Nov 01 22:48:45.150927 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:45.150903 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-csn7v" event={"ID":"8b65bf70-4806-4ed8-b354-614fcbdd0403","Type":"ContainerDied","Data":"cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c"} Nov 01 22:48:45.151159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:45.150949 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb517a75d04644112906b2be95f8973a53be1aadf31c973c1aecee1b358979c" Nov 01 22:48:48.857487 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:48:48.857432 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:48:52.391880 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.391837 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm"] Nov 01 22:48:52.392397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.392315 5036 cpu_manager.go:414] "containerMap: removing container" podUID="8b65bf70-4806-4ed8-b354-614fcbdd0403" containerName="revoke-keys" Nov 01 22:48:52.392397 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.392331 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b65bf70-4806-4ed8-b354-614fcbdd0403" containerName="revoke-keys" Nov 01 22:48:52.392557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.392507 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b65bf70-4806-4ed8-b354-614fcbdd0403" containerName="revoke-keys" Nov 01 22:48:52.395812 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.395786 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.404755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.404595 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz99t\"" Nov 01 22:48:52.404755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.404606 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:52.404755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.404682 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:52.404755 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.404721 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:52.409230 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.409206 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm"] Nov 01 22:48:52.549796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.549758 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.549796 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.549801 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.550037 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.549973 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxzh\" (UniqueName: \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.651101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.651016 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxzh\" (UniqueName: \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.651101 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.651065 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.651317 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.651100 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.651780 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.651755 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.653556 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.653533 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.663686 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.663662 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxzh\" (UniqueName: \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\") pod \"maas-api-revoke-sub-6b19338f-rptwm\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:52.711479 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:52.711430 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:53.032156 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:53.032124 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm"] Nov 01 22:48:53.034100 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:48:53.034070 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38646c87_059f_496b_be7b_0f0a9c28909d.slice/crio-2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52 WatchSource:0}: Error finding container 2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52: Status 404 returned error can't find the container with id 2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52 Nov 01 22:48:53.202975 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:53.202939 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" event={"ID":"38646c87-059f-496b-be7b-0f0a9c28909d","Type":"ContainerStarted","Data":"2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52"} Nov 01 22:48:53.486918 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:53.486831 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:53.624615 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:53.624581 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:54.209941 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:54.209859 5036 generic.go:356] "Generic (PLEG): container finished" podID="38646c87-059f-496b-be7b-0f0a9c28909d" containerID="cc8501de7d96ba7d8009053664650e3500711c12a5e7f3782b259f84ef7b91bc" exitCode=0 Nov 01 22:48:54.210094 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:54.209940 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" event={"ID":"38646c87-059f-496b-be7b-0f0a9c28909d","Type":"ContainerDied","Data":"cc8501de7d96ba7d8009053664650e3500711c12a5e7f3782b259f84ef7b91bc"} Nov 01 22:48:55.346557 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.346531 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:55.377158 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.377132 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\") pod \"38646c87-059f-496b-be7b-0f0a9c28909d\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " Nov 01 22:48:55.377309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.377202 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\") pod \"38646c87-059f-496b-be7b-0f0a9c28909d\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " Nov 01 22:48:55.377309 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.377238 5036 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\" (UniqueName: \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\") pod \"38646c87-059f-496b-be7b-0f0a9c28909d\" (UID: \"38646c87-059f-496b-be7b-0f0a9c28909d\") " Nov 01 22:48:55.377559 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.377531 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca" pod "38646c87-059f-496b-be7b-0f0a9c28909d" (UID: "38646c87-059f-496b-be7b-0f0a9c28909d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:55.379355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.379325 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca" pod "38646c87-059f-496b-be7b-0f0a9c28909d" (UID: "38646c87-059f-496b-be7b-0f0a9c28909d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:55.379355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.379344 5036 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh" pod "38646c87-059f-496b-be7b-0f0a9c28909d" (UID: "38646c87-059f-496b-be7b-0f0a9c28909d"). InnerVolumeSpecName "kube-api-access-bcxzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:55.477991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.477926 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-xks-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:55.477991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.477956 5036 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcxzh\" (UniqueName: \"kubernetes.io/projected/38646c87-059f-496b-be7b-0f0a9c28909d-kube-api-access-bcxzh\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:55.477991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:55.477967 5036 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38646c87-059f-496b-be7b-0f0a9c28909d-maas-api-openshift-ca\") on node \"ip-10-0-131-110.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:56.222023 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:56.221988 5036 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" Nov 01 22:48:56.222215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:56.222021 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rptwm" event={"ID":"38646c87-059f-496b-be7b-0f0a9c28909d","Type":"ContainerDied","Data":"2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52"} Nov 01 22:48:56.222215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:48:56.222082 5036 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eede72d79d4d29276de9993705745278a8ea9a7aed226ce4bc184d4281a4e52" Nov 01 22:49:02.205135 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:02.205102 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:03.246502 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:03.246462 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2grp"] Nov 01 22:49:08.150052 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:08.150022 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:49:08.832364 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:08.832335 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:49:14.104100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:14.104067 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pggxq_6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8/manager/0.log" Nov 01 22:49:14.713658 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:14.713627 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zfclj_31186970-bfbd-4a9d-8923-ddb8582ff2e4/manager/0.log" Nov 01 22:49:14.914300 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:14.914260 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mrv7l_e20c269e-61ab-4af3-b1bf-eba3a120df3a/proxy/0.log" Nov 01 22:49:16.189307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:16.189273 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-csn7v_8b65bf70-4806-4ed8-b354-614fcbdd0403/revoke-keys/0.log" Nov 01 22:49:16.702722 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:16.702687 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rptwm_38646c87-059f-496b-be7b-0f0a9c28909d/revoke-keys/0.log" Nov 01 22:49:17.038488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:17.038463 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m5c79_15be0371-1c7c-4cda-b948-3ba4c23cda59/revoke-keys/0.log" Nov 01 22:49:18.847515 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:18.847484 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvd26_dadece39-3d29-4a88-ace0-e0b9c0d7e2e7/manager/0.log" Nov 01 22:49:18.858628 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:49:18.858598 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:49:19.027415 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:19.027380 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vd5qz_60443ab7-2c7c-4246-aaa1-e6991846441f/manager/0.log" Nov 01 22:49:19.386807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:19.386769 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4hrs_11a55c2e-6d5f-49c8-8482-811ac49cd25b/manager/0.log" Nov 01 22:49:19.575860 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:19.575823 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2grp_d2808eb4-cd97-4607-a8db-8044d728df3d/limitador/0.log" Nov 01 22:49:19.759505 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:19.759403 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t29lh_fe02d9de-6509-43ff-a453-8f9373c24da8/manager/0.log" Nov 01 22:49:20.470227 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:20.470196 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886ft4x6p_ca36d41c-e520-4240-88fa-19e3d4f9386f/istio-proxy/0.log" Nov 01 22:49:22.013284 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:22.013253 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m9bdz_fb61880c-597e-4979-bda6-9f24361aaabd/istio-proxy/0.log" Nov 01 22:49:22.515207 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:22.515177 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-shr8g_ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb/istio-proxy/0.log" Nov 01 22:49:22.695190 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:22.695160 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-tmn2k_90f246de-749e-4ada-81a3-765a18df9fba/istio-proxy/0.log" Nov 01 22:49:22.865534 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:22.865506 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw_a770aa91-03cc-4886-a3a9-3fb96262c2fe/istio-proxy/0.log" Nov 01 22:49:23.379555 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:23.379525 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-554f8bcb97-6mq5w_bd85b460-2a23-473f-98c5-757f7dc2b421/router/0.log" Nov 01 22:49:35.225488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:35.225453 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pggxq_6d80abf6-5dfc-4ea6-8a17-baa20b4da2c8/manager/0.log" Nov 01 22:49:35.818312 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:35.818279 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zfclj_31186970-bfbd-4a9d-8923-ddb8582ff2e4/manager/0.log" Nov 01 22:49:36.026287 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:36.026255 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mrv7l_e20c269e-61ab-4af3-b1bf-eba3a120df3a/proxy/0.log" Nov 01 22:49:36.896215 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:36.896177 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-csn7v_8b65bf70-4806-4ed8-b354-614fcbdd0403/revoke-keys/0.log" Nov 01 22:49:37.400033 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:37.400008 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rptwm_38646c87-059f-496b-be7b-0f0a9c28909d/revoke-keys/0.log" Nov 01 22:49:37.754757 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:37.754666 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m5c79_15be0371-1c7c-4cda-b948-3ba4c23cda59/revoke-keys/0.log" Nov 01 22:49:39.189376 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:39.189343 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvd26_dadece39-3d29-4a88-ace0-e0b9c0d7e2e7/manager/0.log" Nov 01 22:49:39.371238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:39.371202 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vd5qz_60443ab7-2c7c-4246-aaa1-e6991846441f/manager/0.log" Nov 01 22:49:39.720118 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:39.720080 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4hrs_11a55c2e-6d5f-49c8-8482-811ac49cd25b/manager/0.log" Nov 01 22:49:39.887233 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:39.887203 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2grp_d2808eb4-cd97-4607-a8db-8044d728df3d/limitador/0.log" Nov 01 22:49:40.075375 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:40.075346 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t29lh_fe02d9de-6509-43ff-a453-8f9373c24da8/manager/0.log" Nov 01 22:49:40.609706 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:40.609674 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886ft4x6p_ca36d41c-e520-4240-88fa-19e3d4f9386f/istio-proxy/0.log" Nov 01 22:49:42.157264 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:42.157233 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m9bdz_fb61880c-597e-4979-bda6-9f24361aaabd/istio-proxy/0.log" Nov 01 22:49:42.667656 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:42.667621 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-shr8g_ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb/istio-proxy/0.log" Nov 01 22:49:42.825167 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:42.825140 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-tmn2k_90f246de-749e-4ada-81a3-765a18df9fba/istio-proxy/0.log" Nov 01 22:49:42.991086 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:42.991011 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw_a770aa91-03cc-4886-a3a9-3fb96262c2fe/istio-proxy/0.log" Nov 01 22:49:43.483897 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:49:43.483860 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-554f8bcb97-6mq5w_bd85b460-2a23-473f-98c5-757f7dc2b421/router/0.log" Nov 01 22:49:48.856788 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:49:48.856746 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:50:02.266491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:50:02.266426 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:18.856870 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:50:18.856831 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:50:48.856616 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:50:48.856536 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:50:50.678046 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:50:50.678007 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:52.246991 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:50:52.246956 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:02.305205 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:02.305159 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:05.020537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.020498 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gx6ks/must-gather-klzhj"] Nov 01 22:51:05.020972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.020915 5036 cpu_manager.go:414] "containerMap: removing container" podUID="38646c87-059f-496b-be7b-0f0a9c28909d" containerName="revoke-keys" Nov 01 22:51:05.020972 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.020927 5036 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38646c87-059f-496b-be7b-0f0a9c28909d" containerName="revoke-keys" Nov 01 22:51:05.021061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.020994 5036 memory_manager.go:365] "RemoveStaleState removing state" podUID="38646c87-059f-496b-be7b-0f0a9c28909d" containerName="revoke-keys" Nov 01 22:51:05.024187 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.024169 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.028139 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.028115 5036 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gx6ks\"/\"default-dockercfg-8xcsn\"" Nov 01 22:51:05.029070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.029044 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"openshift-service-ca.crt\"" Nov 01 22:51:05.029070 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.029071 5036 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"kube-root-ca.crt\"" Nov 01 22:51:05.044061 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.044036 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/must-gather-klzhj"] Nov 01 22:51:05.050545 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.050513 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:05.050938 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.050918 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-must-gather-output\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.051008 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.050957 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s9z\" (UniqueName: \"kubernetes.io/projected/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-kube-api-access-q8s9z\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.152257 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.152217 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-must-gather-output\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.152398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.152275 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s9z\" (UniqueName: \"kubernetes.io/projected/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-kube-api-access-q8s9z\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.152616 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.152592 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-must-gather-output\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.164551 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.164529 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s9z\" (UniqueName: \"kubernetes.io/projected/ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6-kube-api-access-q8s9z\") pod \"must-gather-klzhj\" (UID: \"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6\") " pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.210414 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.210393 5036 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:05.347521 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.347466 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/must-gather-klzhj" Nov 01 22:51:05.628919 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.628892 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/must-gather-klzhj"] Nov 01 22:51:05.630987 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:51:05.630955 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea0f7ede_0dd9_4ad7_b7c2_81d246ea44e6.slice/crio-502e70708df4b1b108b21c9cf0a9dff8cd454d3fd123050362799c8104cf1605 WatchSource:0}: Error finding container 502e70708df4b1b108b21c9cf0a9dff8cd454d3fd123050362799c8104cf1605: Status 404 returned error can't find the container with id 502e70708df4b1b108b21c9cf0a9dff8cd454d3fd123050362799c8104cf1605 Nov 01 22:51:05.632807 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.632783 5036 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:51:05.899902 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:05.899817 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/must-gather-klzhj" event={"ID":"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6","Type":"ContainerStarted","Data":"502e70708df4b1b108b21c9cf0a9dff8cd454d3fd123050362799c8104cf1605"} Nov 01 22:51:06.908270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:06.907264 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/must-gather-klzhj" event={"ID":"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6","Type":"ContainerStarted","Data":"1adba5700fff7ac45917a3521761d495c91a5f650068db4ffe5804827ed0ef45"} Nov 01 22:51:06.908270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:06.907320 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/must-gather-klzhj" event={"ID":"ea0f7ede-0dd9-4ad7-b7c2-81d246ea44e6","Type":"ContainerStarted","Data":"672b8f948d24c80642aa1f4990defa9c4b9b96cc3ce59889ef29e6b20c4e43b0"} Nov 01 22:51:06.932800 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:06.932731 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gx6ks/must-gather-klzhj" podStartSLOduration=0.975220579 podStartE2EDuration="1.932709386s" totalImagesPullingTime="957.488807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:06.927986785 +0000 UTC m=+4020.620967276" watchObservedRunningTime="2026-11-01 22:51:06.932709386 +0000 UTC m=+4020.625689864" Nov 01 22:51:08.429428 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.429390 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wr8bc_bef85c8b-3d71-48ea-95f5-9104415811b2/global-pull-secret-syncer/0.log" Nov 01 22:51:08.486710 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.486679 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6qtm8_3665c82e-f3a8-461e-bb7a-d4cb6d6111a8/connection-checker/0.log" Nov 01 22:51:08.512723 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.512693 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-chf9m_cef8eaf9-2d56-4d3f-a171-3583becf435e/connection-checker/0.log" Nov 01 22:51:08.540025 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.539997 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lfwr9_cd265d9d-2558-4f1c-8b67-545bf442aa7c/connection-checker/0.log" Nov 01 22:51:08.617336 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.617295 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4tvq4_3cb4e811-66fe-409f-8a75-de7e9e9f936f/konnectivity-agent/0.log" Nov 01 22:51:08.771010 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:08.770979 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-110.us-east-2.compute.internal_fc8a4eb4513497bb3852f6aab7c20b22/haproxy/0.log" Nov 01 22:51:16.430467 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:16.429129 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvd26_dadece39-3d29-4a88-ace0-e0b9c0d7e2e7/manager/0.log" Nov 01 22:51:16.483377 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:16.481033 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vd5qz_60443ab7-2c7c-4246-aaa1-e6991846441f/manager/0.log" Nov 01 22:51:16.609491 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:16.607862 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4hrs_11a55c2e-6d5f-49c8-8482-811ac49cd25b/manager/0.log" Nov 01 22:51:16.635838 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:16.635808 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2grp_d2808eb4-cd97-4607-a8db-8044d728df3d/limitador/0.log" Nov 01 22:51:16.780332 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:16.780293 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t29lh_fe02d9de-6509-43ff-a453-8f9373c24da8/manager/0.log" Nov 01 22:51:17.933100 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:17.933063 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gtrkf_30fdcebb-da92-461b-a1a1-1ab5e46ec0ab/cluster-monitoring-operator/0.log" Nov 01 22:51:17.985739 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:17.985702 5036 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c"] Nov 01 22:51:17.992819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:17.992793 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:17.995914 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:17.995882 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c"] Nov 01 22:51:18.098306 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.098262 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-podres\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.098537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.098436 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-lib-modules\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.098537 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.098511 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885ln\" (UniqueName: \"kubernetes.io/projected/6d8c1a11-402c-4c2b-924d-8792689a95b5-kube-api-access-885ln\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.098688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.098538 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-proc\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.098688 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.098622 5036 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-sys\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.185353 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.185276 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cktms_a68e7dfb-ed88-40cd-b1c9-da441de42fa9/node-exporter/0.log" Nov 01 22:51:18.199548 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199513 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-lib-modules\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199560 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885ln\" (UniqueName: \"kubernetes.io/projected/6d8c1a11-402c-4c2b-924d-8792689a95b5-kube-api-access-885ln\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199595 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-proc\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199643 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-sys\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199695 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199657 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-lib-modules\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199693 5036 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-podres\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199711 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-proc\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199730 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-sys\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.199876 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.199828 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d8c1a11-402c-4c2b-924d-8792689a95b5-podres\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.227138 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.227107 5036 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-885ln\" (UniqueName: \"kubernetes.io/projected/6d8c1a11-402c-4c2b-924d-8792689a95b5-kube-api-access-885ln\") pod \"perf-node-gather-daemonset-fts5c\" (UID: \"6d8c1a11-402c-4c2b-924d-8792689a95b5\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.228342 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.228314 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cktms_a68e7dfb-ed88-40cd-b1c9-da441de42fa9/kube-rbac-proxy/0.log" Nov 01 22:51:18.253889 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.253866 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cktms_a68e7dfb-ed88-40cd-b1c9-da441de42fa9/init-textfile/0.log" Nov 01 22:51:18.330526 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.330488 5036 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:18.624875 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:18.624839 5036 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c"] Nov 01 22:51:18.626330 ip-10-0-131-110 kubenswrapper[5036]: W1101 22:51:18.626300 5036 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d8c1a11_402c_4c2b_924d_8792689a95b5.slice/crio-f093aaf181527b1e19f3611a5275d4f19073a3924b1766f9f6ea55e6cb2329f3 WatchSource:0}: Error finding container f093aaf181527b1e19f3611a5275d4f19073a3924b1766f9f6ea55e6cb2329f3: Status 404 returned error can't find the container with id f093aaf181527b1e19f3611a5275d4f19073a3924b1766f9f6ea55e6cb2329f3 Nov 01 22:51:18.864518 ip-10-0-131-110 kubenswrapper[5036]: E1101 22:51:18.864482 5036 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4bjns" containerName="extractor" Nov 01 22:51:19.084837 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:19.084795 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" event={"ID":"6d8c1a11-402c-4c2b-924d-8792689a95b5","Type":"ContainerStarted","Data":"65064f48f43b892a216e79f8506e9bf34878863c8c2543ff57a466d4a0180ae4"} Nov 01 22:51:19.085270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:19.084864 5036 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" event={"ID":"6d8c1a11-402c-4c2b-924d-8792689a95b5","Type":"ContainerStarted","Data":"f093aaf181527b1e19f3611a5275d4f19073a3924b1766f9f6ea55e6cb2329f3"} Nov 01 22:51:19.085270 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:19.084941 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:19.107159 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:19.107095 5036 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" podStartSLOduration=2.10707615 podStartE2EDuration="2.10707615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:19.102929155 +0000 UTC m=+4032.795909633" watchObservedRunningTime="2026-11-01 22:51:19.10707615 +0000 UTC m=+4032.800056636" Nov 01 22:51:20.128258 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:20.128222 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-5fkz5_521c4405-c6bb-4fd3-bcfb-3cad016376e2/networking-console-plugin/0.log" Nov 01 22:51:20.663468 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:20.660023 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/1.log" Nov 01 22:51:20.664355 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:20.664328 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nscn6_510010af-f8ad-4b48-9f73-e9ad1b0a2de9/console-operator/2.log" Nov 01 22:51:21.885307 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:21.885273 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-q2tv6_490c53c8-ef5b-4296-80b5-32a3204b483f/volume-data-source-validator/0.log" Nov 01 22:51:23.130623 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:23.130578 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55x22_342e82fa-1f59-4454-beb0-d9807dc3d674/dns/0.log" Nov 01 22:51:23.151220 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:23.151187 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55x22_342e82fa-1f59-4454-beb0-d9807dc3d674/kube-rbac-proxy/0.log" Nov 01 22:51:23.282540 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:23.282509 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jxl58_89d6f94c-7d2b-420b-bea6-0bb929dd6194/dns-node-resolver/0.log" Nov 01 22:51:24.076213 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:24.076178 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2cpx_8e6c0a65-773b-45c9-b18b-bd582d7a6307/node-ca/0.log" Nov 01 22:51:25.101955 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:25.101926 5036 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-fts5c" Nov 01 22:51:25.335734 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:25.335684 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886ft4x6p_ca36d41c-e520-4240-88fa-19e3d4f9386f/istio-proxy/0.log" Nov 01 22:51:28.389302 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:28.389261 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m9bdz_fb61880c-597e-4979-bda6-9f24361aaabd/istio-proxy/0.log" Nov 01 22:51:28.477020 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:28.476987 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-shr8g_ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb/istio-proxy/0.log" Nov 01 22:51:28.506238 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:28.506158 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-tmn2k_90f246de-749e-4ada-81a3-765a18df9fba/istio-proxy/0.log" Nov 01 22:51:28.537641 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:28.537601 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw_a770aa91-03cc-4886-a3a9-3fb96262c2fe/istio-proxy/0.log" Nov 01 22:51:28.629596 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:28.629564 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-554f8bcb97-6mq5w_bd85b460-2a23-473f-98c5-757f7dc2b421/router/0.log" Nov 01 22:51:29.033898 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:29.033866 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886ft4x6p_ca36d41c-e520-4240-88fa-19e3d4f9386f/istio-proxy/0.log" Nov 01 22:51:31.519346 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.519311 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m9bdz_fb61880c-597e-4979-bda6-9f24361aaabd/istio-proxy/0.log" Nov 01 22:51:31.573121 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.573089 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-shr8g_ff6c5c17-e832-4a8a-8fe4-31082ba9f7eb/istio-proxy/0.log" Nov 01 22:51:31.594064 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.593997 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-tmn2k_90f246de-749e-4ada-81a3-765a18df9fba/istio-proxy/0.log" Nov 01 22:51:31.614819 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.614788 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-gzknw_a770aa91-03cc-4886-a3a9-3fb96262c2fe/istio-proxy/0.log" Nov 01 22:51:31.671113 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.671065 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-554f8bcb97-6mq5w_bd85b460-2a23-473f-98c5-757f7dc2b421/router/0.log" Nov 01 22:51:31.707681 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:31.707648 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wbmm4_37cb1d59-4690-49c0-95d5-ecf9ce11ff7d/serve-healthcheck-canary/0.log" Nov 01 22:51:32.267174 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.267133 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-lqtc9_2f3b2036-627e-4704-91fb-87df2b35c328/insights-operator/0.log" Nov 01 22:51:32.270562 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.270537 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-lqtc9_2f3b2036-627e-4704-91fb-87df2b35c328/insights-operator/1.log" Nov 01 22:51:32.294419 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.294393 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bjns_6777c192-94cb-4df3-8ed7-3ce7688c0ba7/kube-rbac-proxy/0.log" Nov 01 22:51:32.318488 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.318467 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bjns_6777c192-94cb-4df3-8ed7-3ce7688c0ba7/exporter/0.log" Nov 01 22:51:32.343398 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.343372 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bjns_6777c192-94cb-4df3-8ed7-3ce7688c0ba7/extractor/0.log" Nov 01 22:51:32.535272 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:32.535240 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-vhpgm-lfdc2_f7b9dd27-1e90-4edc-8646-3e7979ef591a/insights-gathering/0.log" Nov 01 22:51:35.054315 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:35.054277 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-86bbk_4b4875ba-473a-4ff3-a1ad-f3a6581c4090/manager/0.log" Nov 01 22:51:35.082666 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:35.082610 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-86bbk_4b4875ba-473a-4ff3-a1ad-f3a6581c4090/copy-manifests/0.log" Nov 01 22:51:35.748878 ip-10-0-131-110 kubenswrapper[5036]: I1101 22:51:35.748837 5036 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886ft4x6p_ca36d41c-e520-4240-88fa-19e3d4f9386f/istio-proxy/0.log"